CheXED: Comparison of a Deep Learning Model to a Clinical Decision Support System for Pneumonia in the Emergency Department

Abstract

Purpose: Patients with pneumonia often present to the emergency department (ED) and require prompt diagnosis and treatment. Clinical decision support systems for the diagnosis and management of pneumonia are commonly utilized in EDs to improve patient care. The purpose of this study is to investigate whether a deep learning model for detecting radiographic pneumonia and pleural effusions can improve functionality of a clinical decision support system (CDSS) for pneumonia management (ePNa) operating in 20 EDs.

Materials and methods: In this retrospective cohort study, a dataset of 7434 prior chest radiographic studies from 6551 ED patients was used to develop and validate a deep learning model to identify radiographic pneumonia, pleural effusions, and evidence of multilobar pneumonia. Model performance was evaluated against 3 radiologists' adjudicated interpretation and compared with performance of the natural language processing of radiology reports used by ePNa.

Results: The deep learning model achieved an area under the receiver operating characteristic curve of 0.833 (95% confidence interval [CI]: 0.795, 0.868) for detecting radiographic pneumonia, 0.939 (95% CI: 0.911, 0.962) for detecting pleural effusions and 0.847 (95% CI: 0.800, 0.890) for identifying multilobar pneumonia. On all 3 tasks, the model achieved higher agreement with the adjudicated radiologist interpretation compared with ePNa.

Conclusions: A deep learning model demonstrated higher agreement with radiologists than the ePNa CDSS in detecting radiographic pneumonia and related findings. Incorporating deep learning models into pneumonia CDSS could enhance diagnostic performance and improve pneumonia management.

Figure 1.. CheXED ROC curves on the test set.

Each plot illustrates the ROC curve (grey line) and operating point (grey diamond) of CheXED. The nonparametric bootstrap with 5,000 replicates was used to estimate the 95% confidence intervals around the performance measures, shown here for the ROC curves (grey region) and operating points (grey dotted line). The reference standard was an adjudication of three radiologists’ interpretations.

Figure 2.. Agreement between CheXED, ePNa, and physician labeling of the radiology report with the reference standard on the test set.

The ePNa CDSS is currently used in 20 emergency departments and uses an NLP system to automatically extract findings from radiology reports. Physician labeling of findings from radiology reports was performed by emergency medicine and pulmonary physicians and was used as supervision for model training. The reference standard was an adjudication of three radiologists’ interpretations of the chest radiographic studies. Weighted Cohen’s Kappa was used to measure agreement between each of the methods and the reference standard, and 95% confidence intervals were estimated using the bootstrap with 5,000 replicates. Asterisks indicate that the agreement with the reference standard is significantly different than CheXED, determined by bootstrapped differences.

Figure 3.. CheXED model interpretation on the test set.

CheXED produced heat maps highlighting the regions of the radiograph which contributed most to its predictions. (a) CheXED incorrectly classified this radiograph as positive for pneumonia, but the opacity in the image was a peripherally calcified breast implant. (b) A consolidation consistent with pneumonia in the left lower lobe was correctly detected by CheXED but missed by the original interpreting radiologist (physician label). (c) A small left-sided pleural effusion was correctly identified by CheXED but not detected by the original interpreting radiologist. (d) The chest radiograph contains a faint consolidation which the CheXED CAM highlights but CheXED didn’t classify this case as pneumonia.