Performance of AI to exclude normal chest radiographs to reduce radiologists' workload

Abstract

Introduction: This study investigates the performance of a commercially available artificial intelligence (AI) system to identify normal chest radiographs and its potential to reduce radiologist workload.

Methods: Retrospective analysis included consecutive chest radiographs from two medical centers between Oct 1, 2016 and Oct 14, 2016. Exclusions comprised follow-up exams within the inclusion period, bedside radiographs, incomplete images, imported radiographs, and pediatric radiographs. Three chest radiologists categorized findings into normal, clinically irrelevant, clinically relevant, urgent, and critical. A commercial AI system processed all radiographs, scoring 10 chest abnormalities on a 0-100 confidence scale. AI system performance was evaluated using the area under the ROC curve (AUC), assessing the detection of normal radiographs. Sensitivity was calculated for the default and a conservative operating point. the detection of negative predictive value (NPV) for urgent and critical findings, as well as the potential workload reduction, was calculated.

Results: A total of 2603 radiographs were acquired in 2141 unique patients. Post-exclusion, 1670 radiographs were analyzed. Categories included 479 normal, 332 clinically irrelevant, 339 clinically relevant, 501 urgent, and 19 critical findings. The AI system achieved an AUC of 0.92. Sensitivity for normal radiographs was 92% at default and 53% at the conservative operating point. At the conservative operating point, NPV was 98% for urgent and critical findings, and could result in a 15% workload reduction.

Conclusion: A commercially available AI system effectively identifies normal chest radiographs and holds the potential to lessen radiologists' workload by omitting half of the normal exams from reporting.

Clinical relevance statement: The AI system is able to detect half of all normal chest radiographs at a clinically acceptable operating point, thereby potentially reducing the workload for the radiologists by 15%.

Key points: The AI system reached an AUC of 0.92 for the detection of normal chest radiographs. Fifty-three percent of normal chest radiographs were identified with a NPV of 98% for urgent findings. AI can reduce the workload of chest radiography reporting by 15%.

Keywords: Artificial intelligence; Diagnosis (computer-assisted); Radiography (thoracic); Respiratory tract diseases; Workload.

Fig. 1

Exclusion flowchart of study participants

Fig. 2

Receiver operating characteristics (ROC) curve for the detection of normal radiographs by the AI system on a consecutive series of chest radiographs from two Dutch medical centers (n = 1670). AUC, area under the curve

Fig. 3

Potential workload reduction compared with the sensitivity of the AI system for normal radiographs (black line) and the sensitivity for urgent/critical findings. The graph shows the tradeoff of increased workload reduction at the cost less sensitivity for urgent/critical findings. The maximum workload reduction is 1670/2080 (analyzed radiographs/analyzed radiographs + excluded radiographs) = 80.3%. Higher workload reduction (gray area) could not be achieved in this setup

Fig. 4

The four cases with urgent findings missed by the AI system at the conservative operating point. a From earlier CT known stable nodule in the right lower quadrant, with characteristics of a subpleural lymph node on CT. Classified as an urgent finding by two of the three radiologists. b CT confirmed left hilar mass. Classified as an urgent finding by all three radiologists. c Suspicion of minor post-tuberculosis fibrotic changes in the upper right lung. Classified as an urgent finding by two of the three radiologists. d In the original report description of small right paracardial infiltrate. Classified as an urgent finding by two of the three radiologists