A comparison of Covid-19 early detection between convolutional neural networks and radiologists

Abstract

Background: The role of chest radiography in COVID-19 disease has changed since the beginning of the pandemic from a diagnostic tool when microbiological resources were scarce to a different one focused on detecting and monitoring COVID-19 lung involvement. Using chest radiographs, early detection of the disease is still helpful in resource-poor environments. However, the sensitivity of a chest radiograph for diagnosing COVID-19 is modest, even for expert radiologists. In this paper, the performance of a deep learning algorithm on the first clinical encounter is evaluated and compared with a group of radiologists with different years of experience.

Methods: The algorithm uses an ensemble of four deep convolutional networks, Ensemble4Covid, trained to detect COVID-19 on frontal chest radiographs. The algorithm was tested using images from the first clinical encounter of positive and negative cases. Its performance was compared with five radiologists on a smaller test subset of patients. The algorithm's performance was also validated using the public dataset COVIDx.

Results: Compared to the consensus of five radiologists, the Ensemble4Covid model achieved an AUC of 0.85, whereas the radiologists achieved an AUC of 0.71. Compared with other state-of-the-art models, the performance of a single model of our ensemble achieved nonsignificant differences in the public dataset COVIDx.

Conclusion: The results show that the use of images from the first clinical encounter significantly drops the detection performance of COVID-19. The performance of our Ensemble4Covid under these challenging conditions is considerably higher compared to a consensus of five radiologists. Artificial intelligence can be used for the fast diagnosis of COVID-19.

Keywords: Covid-19; Deep learning; Radiology.

Fig. 1

Flowchart for data curation and data partition

Fig. 2

Grad-CAM heatmaps for True Positive COVID-19 predictions. The red intensity on the right images indicates the image regions that are important for the positive prediction. These regions correspond to patchy consolidations and ground-glass opacities, with a peripheral, bilateral distribution (a) or a mid and lower zone predominance on the right lung (b, c)

Fig. 3

Grad-CAM heatmaps for False Positive COVID-19 predictions. The red intensity of the images on the right indicates the important regions for a false positive prediction. These regions correspond to multiple consolidations with no zone predominance in the right lung (a), a unilateral right lower lung opacity and the contralateral soft tissues in the axillary region (b), superimposition of the scapula in the right lung periphery along with the left lower lobe vessels behind the heart (c)

Fig. 4

Sensitivity and specificity for each radiologist with 95% confidence intervals with the ROC curves for the Ensemble4Model (with 95% confidence bands) and the five radiologists' consensus. Each reader shows two operating points depending on the threshold over the scores