Radiographic findings in COVID-19: Comparison between AI and radiologist

Abstract

Context: As the burden of COVID-19 enhances, the need of a fast and reliable screening method is imperative. Chest radiographs plays a pivotal role in rapidly triaging the patients. Unfortunately, in low-resource settings, there is a scarcity of trained radiologists.

Aim: This study evaluates and compares the performance of an artificial intelligence (AI) system with a radiologist in detecting chest radiograph findings due to COVID-19.

Subjects and methods: The test set consisted of 457 CXR images of patients with suspected COVID-19 pneumonia over a period of three months. The radiographs were evaluated by a radiologist with experience of more than 13 years and by the AI system (NeuraCovid, a web application that pairs with the AI model COVID-NET). Performance of AI system and the radiologist were compared by calculating the sensitivity, specificity and generating a receiver operating characteristic curve. RT-PCR test results were used as the gold standard.

Results: The radiologist obtained a sensitivity and specificity of 44.1% and 92.5%, respectively, whereas the AI had a sensitivity and specificity of 41.6% and 60%, respectively. The area under curve for correctly classifying CXR images as COVID-19 pneumonia was 0.48 for the AI system and 0.68 for the radiologist. The radiologist's prediction was found to be superior to that of the AI with a P VALUE of 0.005.

Conclusion: The specificity and sensitivity of detecting lung involvement in COVID-19, by the radiologist, was found to be superior to that by the AI system.

Keywords: Artificial intelligence; COVID pneumonia; chest radiographs; rapid triaging.

Figure 1 (A-C)

This figure demonstrates the inference generated by the AI system in a 48-year-old RT-PCR positive male patient. (A) Plain PA chest radiograph of the patient shows multiple areas of inhomogeneous opacities predominantly in the peripheral zones (arrows) and was classified as COVID pneumonia by the radiologist. (B) Heat Map image of the corresponding radiograph showing areas of involvement. (C) Inference report generated by the AI system predicting it as COVID pneumonia with a 96% likelihood ratio. Note the geographic and opacity severity scores generated by the system

Figure 2 (A-C)

(A) PA chest radiograph of a 9-year-old RT-PCR negative male patient shows no significant lung abnormality. (B) The Heat Map image of the corresponding radiograph shows no infected areas in both the lung fields. (C) The inference generated by the AI system predicts the radiograph as normal with a 98% likelihood percentage

Figure 3 (A-C)

(A) Shows a PA chest radiograph of a 30-year-old RT-PCR negative female patient with no obvious lung opacities (labelled as negative by the radiologist). (B) The misinterpreted Heat Map image of the corresponding radiograph shows infected areas which is seen the right upper and mid zone and outside the lung fields. (C) The inference generated by the AI system predicts the normal radiograph as COVID with a likelihood of 89% (false positive)

Figure 4 (A-C)

(A): PA chest radiograph of a 50-year-old RT-PCR positive female patient with peripheral subpleural lung opacities (arrows) in both the lower zones which was classified as COVID positive by the radiologist. (B): The Heat Map image of the corresponding radiograph shows no areas of lung involvement. (C): The inference generated by the AI system predicts the radiograph as normal with a likelihood of 61% (false negative)

Figure 5

Diagnostic test comparison of radiologist with gold standard RT-PCR

Figure 6

Diagnostic test comparison of AI with gold standard RT-PCR

Figure 7

Shows the agreement between the gold standard and AI