Leveraging Deep Learning Decision-Support System in Specialized Oncology Center: A Multi-Reader Retrospective Study on Detection of Pulmonary Lesions in Chest X-ray Images

Abstract

Chest X-ray (CXR) is considered to be the most widely used modality for detecting and monitoring various thoracic findings, including lung carcinoma and other pulmonary lesions. However, X-ray imaging shows particular limitations when detecting primary and secondary tumors and is prone to reading errors due to limited resolution and disagreement between radiologists. To address these issues, we developed a deep-learning-based automatic detection algorithm (DLAD) to automatically detect and localize suspicious lesions on CXRs. Five radiologists were invited to retrospectively evaluate 300 CXR images from a specialized oncology center, and the performance of individual radiologists was subsequently compared with that of DLAD. The proposed DLAD achieved significantly higher sensitivity (0.910 (0.854-0.966)) than that of all assessed radiologists (RAD 10.290 (0.201-0.379), p < 0.001, RAD 20.450 (0.352-0.548), p < 0.001, RAD 30.670 (0.578-0.762), p < 0.001, RAD 40.810 (0.733-0.887), p = 0.025, RAD 50.700 (0.610-0.790), p < 0.001). The DLAD specificity (0.775 (0.717-0.833)) was significantly lower than for all assessed radiologists (RAD 11.000 (0.984-1.000), p < 0.001, RAD 20.970 (0.946-1.000), p < 0.001, RAD 30.980 (0.961-1.000), p < 0.001, RAD 40.975 (0.953-0.997), p < 0.001, RAD 50.995 (0.985-1.000), p < 0.001). The study results demonstrate that the proposed DLAD could be utilized as a decision-support system to reduce radiologists' false negative rate.

Keywords: YOLO; computer-aided diagnosis; convolutional neural network; deep learning; lung cancer; object detection; pulmonary lesion.

Figure A1

False negative (FN) images incorrectly classified by DLAD software during the retrospective study as true negative (TN) and the respected ground truth pixel-level annotation. Assessed radiologist RAD 1 also incorrectly classified CXR images #1, #2, #4, #5, #6, #7, #8 and #9. Assessed radiologist RAD 2 also incorrectly classified CXR images #1, #2, #3, #4, #5, #6 and #8. Assessed radiologist RAD 3 also incorrectly classified CXR images #1, #2, #3, #4, #5, #6 and #8. Assessed radiologist RAD 4 also incorrectly classified CXR images #1, #2, #4, and #8. Assessed radiologist RAD 5 also incorrectly classified CXR images #2, #4, #5, #6 and #9.

Figure A2

Confusion matrix showing the performance of the proposed DLAD and individual radiologists. All of them incorrectly evaluated some assessed LES+ Abnormal CXRs as without suspicious lesions. For individual radiologists, it was 62, 46, 24, 10 and 21 images, which the assessed radiologists would check again and/or consult with a more experienced colleague.

Figure 1

Initial and follow-up CXRs and CT images with a pulmonary lesion. (a) Initial CXR of a 65-year-old male patient with metastatic renal cell carcinoma in the left upper lobe and (b) CT examination of the patient (a). (c) Follow-up CXR of (a). (d) Initial CXR of an 81-year-old male patient with metastatic adenocarcinoma in the right middle lobe and (e) CT examination corresponding to (d). (f) Follow-up CXR of (d). The yellow arrows indicate the localization of suspected pulmonary lesions [16].

Figure 2

(a) The proposed DLAD (Carebot AI CXR v2.00, implemented in CloudPACS by OR-CZ) and other commercial solutions: (b) Qure AI qXR, (c) Lunit INSIGHT CXR, and (d) Arterys Chest AI.

Figure 3

Overview of the YOLOv5 model architecture [34].

Figure 4

Examples of the respected ground truth pixel-level annotations and correct DLAD predictions (TP). The proposed DLAD correctly identified 91 out of 100 CXRs (Se of 0.910 (0.854–0.966)) pulmonary lesions (LES+ Abnormal).

Figure 5

Forest plots showing the mean sensitivity (Se), specificity (Sp), positive (PLR), and negative likelihood ratio (NLR) and corresponding 95% confidence interval estimates for DLAD and individual radiologists. For all assessed radiologists, the DLAD achieved a statistically significantly higher Se than that of radiologists, indicating that it would be useful in identifying patients with pulmonary lesions that were not identified by the radiologists.