Clinical utility of an artificial intelligence radiomics-based tool for risk stratification of pulmonary nodules

Abstract

Background: Clinical utility data on pulmonary nodule (PN) risk stratification biomarkers are lacking. We aimed to determine the incremental predictive value and clinical utility of using an artificial intelligence (AI) radiomics-based computer-aided diagnosis (CAD) tool in addition to routine clinical information to risk stratify PNs among real-world patients.

Methods: We performed a retrospective cohort study of patients with PNs who underwent lung biopsy. We collected clinical data and used a commercially available AI radiomics-based CAD tool to calculate a Lung Cancer Prediction (LCP) score. We developed logistic regression models to evaluate a well-validated clinical risk prediction model (the Mayo Clinic model) with and without the LCP score (Mayo vs Mayo + LCP) using area under the curve (AUC), risk stratification table, and standardized net benefit analyses.

Results: Among the 134 patients undergoing PN biopsy, cancer prevalence was 61%. Addition of the radiomics-based LCP score to the Mayo model was associated with increased predictive accuracy (likelihood ratio test, P = .012). The AUCs for the Mayo and Mayo + LCP models were 0.58 (95% CI = 0.48 to 0.69) and 0.65 (95% CI = 0.56 to 0.75), respectively. At the 65% risk threshold, the Mayo + LCP model was associated with increased sensitivity (56% vs 38%; P = .019), similar false positive rate (33% vs 35%; P = .8), and increased standardized net benefit (18% vs -3.3%) compared with the Mayo model.

Conclusions: Use of a commercially available AI radiomics-based CAD tool as a supplement to clinical information improved PN cancer risk prediction and may result in clinically meaningful changes in risk stratification.

Figure 1.

Assembly of the analytic sample. CT = computed tomography; LCP-CAD = Lung Cancer Prediction Computer-Aided Diagnosis.

Figure 2.

Calibration belt plots with 80% and 95% confidence levels for the A) Mayo model and B) the Mayo + LCP model. In each graph, the red line represents perfect correlation between expected and observed malignancy probabilities. For each model, the Wald test was used to evaluate overall calibration quality by testing the null hypothesis of no statistically significant difference between the observed and expected probabilities. LCP = Lung Cancer Prediction.

Figure 3.

Reclassification analysis assessing the theoretical clinical utility of supplementing clinical risk assessment with the LCP score. A) Among the 82 patients with malignant pulmonary nodules, 51 (62%) were classified as intermediate risk (5%-65%) by the Mayo model. Nineteen (37%) of these patients were reclassified to high risk (>65%) with the Mayo + LCP model. B) Among the 52 patients with benign pulmonary nodules, 18 (35%) were classified as high risk (>65%) by the Mayo model. Eight (44%) of these patients were reclassified to intermediate risk (5%-65%) by the Mayo + LCP model.

Figure 4.

Reclassification plots for the Mayo vs Mayo + LCP models at the 65% risk threshold, stratified by A) malignant and B) benign PN final diagnosis. Among the 51 malignant PNs classified as <65% by the Mayo model, 19 (37%) were reclassified as >65% by the Mayo + LCP model. Among the 31 malignant PNs classified as >65% by the Mayo model, 4 (13%) were reclassified as <65% by the Mayo + LCP model. Among the 18 benign PNs classified as >65% by the Mayo model, 8 (44%) were reclassified as <65% by the Mayo + LCP model. Among the 34 benign PNs classified as <65% by the Mayo model, 7 (21%) were reclassified as >65% by the Mayo + LCP model.