Automated diagnosis and prognosis of COVID-19 pneumonia from initial ER chest X-rays using deep learning

Abstract

Background: Airspace disease as seen on chest X-rays is an important point in triage for patients initially presenting to the emergency department with suspected COVID-19 infection. The purpose of this study is to evaluate a previously trained interpretable deep learning algorithm for the diagnosis and prognosis of COVID-19 pneumonia from chest X-rays obtained in the ED.

Methods: This retrospective study included 2456 (50% RT-PCR positive for COVID-19) adult patients who received both a chest X-ray and SARS-CoV-2 RT-PCR test from January 2020 to March of 2021 in the emergency department at a single U.S.

Institution: A total of 2000 patients were included as an additional training cohort and 456 patients in the randomized internal holdout testing cohort for a previously trained Siemens AI-Radiology Companion deep learning convolutional neural network algorithm. Three cardiothoracic fellowship-trained radiologists systematically evaluated each chest X-ray and generated an airspace disease area-based severity score which was compared against the same score produced by artificial intelligence. The interobserver agreement, diagnostic accuracy, and predictive capability for inpatient outcomes were assessed. Principal statistical tests used in this study include both univariate and multivariate logistic regression.

Results: Overall ICC was 0.820 (95% CI 0.790-0.840). The diagnostic AUC for SARS-CoV-2 RT-PCR positivity was 0.890 (95% CI 0.861-0.920) for the neural network and 0.936 (95% CI 0.918-0.960) for radiologists. Airspace opacities score by AI alone predicted ICU admission (AUC = 0.870) and mortality (0.829) in all patients. Addition of age and BMI into a multivariate log model improved mortality prediction (AUC = 0.906).

Conclusion: The deep learning algorithm provides an accurate and interpretable assessment of the disease burden in COVID-19 pneumonia on chest radiographs. The reported severity scores correlate with expert assessment and accurately predicts important clinical outcomes. The algorithm contributes additional prognostic information not currently incorporated into patient management.

Keywords: COVID-19; Critical care; Deep learning; Pulmonology; Radiology.

Fig. 1

Flow-diagram describing inclusion of patients for COVID-19 training and test datasets. 23,805 X-rays were queried and ultimately 2488 met criteria of a documented COVID-19 test within 14 days of an eligible PA or AP CXR. 2000 were used in the training cohort with 488 retained as internal holdout for validation. Missing data from 32 patients, defined as images that failed the AI segmentation due to poor imaging quality, were excluded

Fig. 2

Visual representation of neural network annotations and outputs. A AP portable CXR with left lower lobe airspace opacities scored a 4/10 by the dCNN. EKG leads overlie the chest bilaterally. B Upright portable AP view CXR with bilateral airspace opacities scored an 8/10 by the dCNN. Dual chamber pacemaker with atrial and ventricular leads overlies the left chest. C dCNNs architecture used for classification and detection of airspace opacities. A ResNet backbone for the image anatomy feeds forward into a voxel feature pyramid which is then forwarded to a convolutional network-based detector for classification of the airspace opacity. A detailed description of the architecture can be found in the materials and methods under Deep Convolutional Neural Network Algorithm

Fig. 3

Prediction of Positive SARS-CoV-2 PCR by extent of AI-determined airspace disease. A Logistic probability plot of positive SARS-CoV-2 PCR as a function of AI-determined airspace extent. Median airspace extent (40%) had just under 50% probability of a concurrent positive PCR. McFadden R² = 0.412. B ROC curve for prediction of SARS-CoV-2 PCR positivity in comparison to radiologist impression of airspace extent. Radiologist (AUC = 0.936, 95% CI 0.918–0.960) and AI (AUC = 0.890, 95% CI 0.861–0.920) annotations were both highly accurate

Fig. 4

Comparison of differences between AI and Radiologist measurement of airspace opacity extent. A Airspace opacity extent percentage as a function of observer. Adjusted R² = 0.656; Spearman ρ = 0.797. Overall agreement is considered excellent for positive cases (single fixed raters ICC = 0.810, 95% CI 0.765–0.840). Agreement for all cases is considered excellent (single fixed raters ICC = 0.820, 95% CI 0.790–0.840). B Bland–Altman plot for difference of methods. Mean difference -22.4%; SE 21.1%. C Confusion matrix for discrete scores compared between expert and AI. Weighted macro F1 score for categorical agreement is 0.157

Fig. 5

Prediction of outcomes by use of AI-determined airspace opacity extent (AI-ASOS) using simple logistic regression. A Prediction of outcomes in all patients. AI-ASOS is best at predicting ICU admission (AUC = 0.870, 95% CI 0.834–0.904) and pulmonary mortality (AUC = 0.845, 95% CI 0.802–0.888). B Prediction of outcomes statistics amongst all patients using a multivariate empirically derived model of additional clinical risk factors. Use of AI-ASOS, age, and BMI had a high accuracy for prediction of mortality statistics and ICU admission (AUC = 0.906, 0.896, and 0.880, respectively)

Fig. 6

Probabilities of outcomes as a function of AI-determined airspace opacity extent (AI-ASOS). A Probability of ICU admission. 50% airspace opacity extent (AI-ASOS = 5) confers a ~ 20% chance of ICU admission. B Probability of pulmonary death. Risk of pulmonary death begins increasing at roughly 50% airspace opacity extent (AI-ASOS = 5)