Comparison of AI software tools for automated detection, quantification and categorization of pulmonary nodules in the HANSE LCS trial

Abstract

Participant management in a lung cancer screening (LCS) depends on the assigned Lung Imaging Reporting and Data System (Lung-RADS) category, which is based on reliable detection and measurement of pulmonary nodules. The aim of this study was to compare the agreement of two AI-based software tools for detection, quantification and categorization of pulmonary nodules in an LCS program in Northern Germany (HANSE-trial). 946 low-dose baseline CT-examinations were analyzed by two AI software tools regarding lung nodule detection, quantification and categorization and compared to the final radiologist read. The relationship between detected nodule volumes by both software tools was assessed by Pearson correlation (r) and tested for significance using Wilcoxon signed-rank test. The consistency of Lung-RADS classifications between Software tool 1 (S1, Aview v2.5, Coreline Soft, Seoul, Korea) and Software tool 2 (S2, Prototype ''ChestCTExplore'', software version ToDo, Siemens Healthineers, Forchheim, Germany) was evaluated by Cohen's kappa (κ) and percentual agreement (PA).The derived volumes of true positive nodules were strongly correlated (r > 0.95), the volume derived by S2 was significantly higher than by S1 (P < 0.0001, mean difference: 6mm³). Moderate PA (62%) between S1 and S2 was found in the assignment of Lung-RADS classification (κ = 0.45). The PA of Lung-RADS classification to final read was 75% and 55% for S1 and S2, but the incorporation of S1 into the initial nodule detection and segmentation must be considered here. Significant nodule volume differences between AI software tools lead to different Lung-RADS scores in 38% of cases, which may result in altered participant management. Therefore, high performance and agreement of accredited AI software tools are necessary for a future national LCS program.

Fig. 1

Examples of FP detection for both software tools. In (a) a consolidation due to pneumonia, in (b) a metallic foreign body and in (c) an osteophyte cases were detected by both software tools as a pulmonary nodule.

Fig. 2

Exemplary FN detection of pulmonary solid nodule (volume 1898.2 mm³, Lung-RADS 4B category) for both software tools for a 67-year-old female patient. No malignancy was found in the biopsy.

Fig. 3

Exemplary FN detection of pulmonary solid mass (volume 5976.7 mm³, Lung-RADS 4X category) for S1 software tool for a 69-year-old patient. After CT-assisted transthoracic puncture, the histology showed adenocarcinoma.

Fig. 4

Bland-Altman analysis of volume for all nodules (A), for nodules with volume ≥ 34 mm³ (B) and nodules with volume ≥ 113 mm³ (C) between software tools S1 and S2. The full red line indicates the mean difference (-13.3 (A), -6.0 (B) and 31.6 (C) mm³) and blue dotted lines indicate 95% limits of agreement.

Fig. 5

Bland-Altman analysis of volume for all nodules (A), for nodules with volume ≥ 34 mm³ (B) and for nodules with volume ≥ 113 mm³ (C) between FR and software tool S2. The full red line indicates the mean difference (-15.7 (A), -7.6 (B) and 29.7 (C) mm³) and blue dotted lines indicate 95% limits of agreement.

Fig. 6

Bland-Altman analysis of volume for all nodules (A), for nodules with volume ≥ 34 mm³ (B) and for nodules with volume ≥ 113 mm³ (C) between FR and software tool S1. The full red line indicates the mean difference (9.5 (A), 22.6 (B) and 106.4 (C) mm³) and blue dotted lines indicate 95% limits of agreement.