Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy

Abstract

Background Coronavirus disease 2019 (COVID-19) has widely spread all over the world since the beginning of 2020. It is desirable to develop automatic and accurate detection of COVID-19 using chest CT. Purpose To develop a fully automatic framework to detect COVID-19 using chest CT and evaluate its performance. Materials and Methods In this retrospective and multicenter study, a deep learning model, the COVID-19 detection neural network (COVNet), was developed to extract visual features from volumetric chest CT scans for the detection of COVID-19. CT scans of community-acquired pneumonia (CAP) and other non-pneumonia abnormalities were included to test the robustness of the model. The datasets were collected from six hospitals between August 2016 and February 2020. Diagnostic performance was assessed with the area under the receiver operating characteristic curve, sensitivity, and specificity. Results The collected dataset consisted of 4352 chest CT scans from 3322 patients. The average patient age (±standard deviation) was 49 years ± 15, and there were slightly more men than women (1838 vs 1484, respectively; P = .29). The per-scan sensitivity and specificity for detecting COVID-19 in the independent test set was 90% (95% confidence interval [CI]: 83%, 94%; 114 of 127 scans) and 96% (95% CI: 93%, 98%; 294 of 307 scans), respectively, with an area under the receiver operating characteristic curve of 0.96 (P < .001). The per-scan sensitivity and specificity for detecting CAP in the independent test set was 87% (152 of 175 scans) and 92% (239 of 259 scans), respectively, with an area under the receiver operating characteristic curve of 0.95 (95% CI: 0.93, 0.97). Conclusion A deep learning model can accurately detect coronavirus 2019 and differentiate it from community-acquired pneumonia and other lung conditions. © RSNA, 2020 Online supplemental material is available for this article.

Fig 1.

Flow diagram. We collected a dataset of 3506 patients with chest CT exams. After exclusion, 3,322 eligible patients were included for the model development and evaluation in this study. CT exams were extracted from DICOM files. The dataset was split into a training set (to training the model), and the independent testing set at the patient level. A supervised deep learning framework (COVNet) was developed to detect COVID-19 and community acquired pneumonia. The predictive performance of the model was evaluated by using an independent testing set. COVNet = COVID-19 detection neural network.

Fig 2.

COVID-19 detection neural network (COVNet) architecture. The COVNet is a convolutional neural network (CNN) using ResNet50 as the backbone. It takes as input a series of CT slices and generates a classification prediction of the CT image. The CNN features from each slice of the CT series are combined by a max-pooling operation and the resulting feature map is fed to a fully connected layer to generate a probability score for each class.

Fig 3a.

Receiver operating characteristic curves of the model. Each plot illustrates the receiver operating characteristic (ROC) curve of the algorithm (black curve) on the independent testing set for (a) COVID-19 with AUC = 0.96 (p-value<0.001) (b) CAP with AUC = 0.95 (p-value<0.001) and (c) Non-Pneumonia with AUC = 0.98 (p-value<0.001). The gray region indicates the 95% CI. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia, CI=confidence interval.

Fig 3b.

Receiver operating characteristic curves of the model. Each plot illustrates the receiver operating characteristic (ROC) curve of the algorithm (black curve) on the independent testing set for (a) COVID-19 with AUC = 0.96 (p-value<0.001) (b) CAP with AUC = 0.95 (p-value<0.001) and (c) Non-Pneumonia with AUC = 0.98 (p-value<0.001). The gray region indicates the 95% CI. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia, CI=confidence interval.

Fig 3c.

Receiver operating characteristic curves of the model. Each plot illustrates the receiver operating characteristic (ROC) curve of the algorithm (black curve) on the independent testing set for (a) COVID-19 with AUC = 0.96 (p-value<0.001) (b) CAP with AUC = 0.95 (p-value<0.001) and (c) Non-Pneumonia with AUC = 0.98 (p-value<0.001). The gray region indicates the 95% CI. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia, CI=confidence interval.

Fig 4a.

Representative examples of the attention heatmaps generated using Grad-CAM method for (a) COVID-19, (b) CAP, and (c) Non-Pneumonia. The heatmaps are standard Jet colormap and overlapped on the original image, the red color highlights the activation region associated with the predicted class. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia.

Fig 4b.

Representative examples of the attention heatmaps generated using Grad-CAM method for (a) COVID-19, (b) CAP, and (c) Non-Pneumonia. The heatmaps are standard Jet colormap and overlapped on the original image, the red color highlights the activation region associated with the predicted class. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia.

Fig 4c.

Representative examples of the attention heatmaps generated using Grad-CAM method for (a) COVID-19, (b) CAP, and (c) Non-Pneumonia. The heatmaps are standard Jet colormap and overlapped on the original image, the red color highlights the activation region associated with the predicted class. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia.

Fig 5.

A representative example of community acquired pneumonia case that is misclassified as COVID-19. The consecutive slices around the abnormality are shown from left to right. COVID-19 = coronavirus disease 2019, CAP = community acquired pneumonia.

Fig 6.

A representative example of COVID-19 case that is misclassified as community acquired pneumonia. The consecutive slices around the abnormality are shown from left to right. COVID-19 = coronavirus disease 2019.