Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT

Abstract

Background Limited data are available regarding whether computer-aided diagnosis (CAD) improves assessment of malignancy risk in indeterminate pulmonary nodules (IPNs). Purpose To evaluate the effect of an artificial intelligence-based CAD tool on clinician IPN diagnostic performance and agreement for both malignancy risk categories and management recommendations. Materials and Methods This was a retrospective multireader multicase study performed in June and July 2020 on chest CT studies of IPNs. Readers used only CT imaging data and provided an estimate of malignancy risk and a management recommendation for each case without and with CAD. The effect of CAD on average reader diagnostic performance was assessed using the Obuchowski-Rockette and Dorfman-Berbaum-Metz method to calculate estimates of area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Multirater Fleiss κ statistics were used to measure interobserver agreement for malignancy risk and management recommendations. Results A total of 300 chest CT scans of IPNs with maximal diameters of 5-30 mm (50.0% malignant) were reviewed by 12 readers (six radiologists, six pulmonologists) (patient median age, 65 years; IQR, 59-71 years; 164 [55%] men). Readers' average AUC improved from 0.82 to 0.89 with CAD (P < .001). At malignancy risk thresholds of 5% and 65%, use of CAD improved average sensitivity from 94.1% to 97.9% (P = .01) and from 52.6% to 63.1% (P < .001), respectively. Average reader specificity improved from 37.4% to 42.3% (P = .03) and from 87.3% to 89.9% (P = .05), respectively. Reader interobserver agreement improved with CAD for both the less than 5% (Fleiss κ, 0.50 vs 0.71; P < .001) and more than 65% (Fleiss κ, 0.54 vs 0.71; P < .001) malignancy risk categories. Overall reader interobserver agreement for management recommendation categories (no action, CT surveillance, diagnostic procedure) also improved with CAD (Fleiss κ, 0.44 vs 0.52; P = .001). Conclusion Use of computer-aided diagnosis improved estimation of indeterminate pulmonary nodule malignancy risk on chest CT scans and improved interobserver agreement for both risk stratification and management recommendations. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Yanagawa in this issue.

Graphical abstract

Figure 1:

The lung cancer prediction score is generated by an artificial intelligence tool and categorizes pulmonary nodule malignancy risk on a decile scale, with a score of 1 representing nodules at lowest risk and a score of 10 indicating nodules at highest risk.

Figure 2:

Flowchart shows inclusion and exclusion criteria for pulmonary nodules included in the study. IPNs = indeterminate pulmonary nodules.

Figure 3:

Representative axial CT images of pulmonary nodules included in the study. (A) Malignant nodule with a lung cancer prediction score of 10. (B) Benign nodule with a lung cancer prediction score of 2.

Figure 4:

Average reader receiver operating characteristic curves for discrimination of indeterminate pulmonary nodules under two reading conditions: without computer-aided diagnosis (CAD) and with CAD. Average area under the receiver operating characteristic curve (AUC) was computed across 12 readers participating in the study using either the Obuchowski-Rockette and Dorfman-Berbaum-Metz method, which accounts for the multireader multicase study design.

Figure 5:

Individual reader discrimination under two reading conditions: without computer-aided diagnosis (CAD) and with CAD. There was a significant improvement in area under the receiver operating characteristic curve (AUC) for each reader (P ≤ .001) with CAD.

Figure 6:

Reclassification plots with and without computer-aided diagnosis (CAD) for malignant and benign pulmonary nodules. Summary plots of all pairs of pre-CAD (x-axis) and post-CAD (y-axis) malignancy risk estimates for malignant (n = 1800 [150 cases × 12 readers]) (A) and benign (n = 1800 [150 cases × 12 readers]) (B) nodules. Malignancy risk decision thresholds of 5% and 65% are depicted as gray lines in each plot.