Automated Artificial Intelligence Model Trained on a Large Data Set Can Detect Pancreas Cancer on Diagnostic Computed Tomography Scans As Well As Visually Occult Preinvasive Cancer on Prediagnostic Computed Tomography Scans

Abstract

Background & aims: The aims of our case-control study were (1) to develop an automated 3-dimensional (3D) Convolutional Neural Network (CNN) for detection of pancreatic ductal adenocarcinoma (PDA) on diagnostic computed tomography scans (CTs), (2) evaluate its generalizability on multi-institutional public data sets, (3) its utility as a potential screening tool using a simulated cohort with high pretest probability, and (4) its ability to detect visually occult preinvasive cancer on prediagnostic CTs.

Methods: A 3D-CNN classification system was trained using algorithmically generated bounding boxes and pancreatic masks on a curated data set of 696 portal phase diagnostic CTs with PDA and 1080 control images with a nonneoplastic pancreas. The model was evaluated on (1) an intramural hold-out test subset (409 CTs with PDA, 829 controls); (2) a simulated cohort with a case-control distribution that matched the risk of PDA in glycemically defined new-onset diabetes, and Enriching New-Onset Diabetes for Pancreatic Cancer score ≥3; (3) multi-institutional public data sets (194 CTs with PDA, 80 controls), and (4) a cohort of 100 prediagnostic CTs (i.e., CTs incidentally acquired 3-36 months before clinical diagnosis of PDA) without a focal mass, and 134 controls.

Results: Of the CTs in the intramural test subset, 798 (64%) were from other hospitals. The model correctly classified 360 CTs (88%) with PDA and 783 control CTs (94%), with a mean accuracy 0.92 (95% CI, 0.91-0.94), area under the receiver operating characteristic (AUROC) curve of 0.97 (95% CI, 0.96-0.98), sensitivity of 0.88 (95% CI, 0.85-0.91), and specificity of 0.95 (95% CI, 0.93-0.96). Activation areas on heat maps overlapped with the tumor in 350 of 360 CTs (97%). Performance was high across tumor stages (sensitivity of 0.80, 0.87, 0.95, and 1.0 on T1 through T4 stages, respectively), comparable for hypodense vs isodense tumors (sensitivity: 0.90 vs 0.82), different age, sex, CT slice thicknesses, and vendors (all P > .05), and generalizable on both the simulated cohort (accuracy, 0.95 [95% 0.94-0.95]; AUROC curve, 0.97 [95% CI, 0.94-0.99]) and public data sets (accuracy, 0.86 [95% CI, 0.82-0.90]; AUROC curve, 0.90 [95% CI, 0.86-0.95]). Despite being exclusively trained on diagnostic CTs with larger tumors, the model could detect occult PDA on prediagnostic CTs (accuracy, 0.84 [95% CI, 0.79-0.88]; AUROC curve, 0.91 [95% CI, 0.86-0.94]; sensitivity, 0.75 [95% CI, 0.67-0.84]; and specificity, 0.90 [95% CI, 0.85-0.95]) at a median 475 days (range, 93-1082 days) before clinical diagnosis.

Conclusions: This automated artificial intelligence model trained on a large and diverse data set shows high accuracy and generalizable performance for detection of PDA on diagnostic CTs as well as for visually occult PDA on prediagnostic CTs. Prospective validation with blood-based biomarkers is warranted to assess the potential for early detection of sporadic PDA in high-risk individuals.

Keywords: Artificial Intelligence; Biomarkers; Computed Tomography; Pancreas; Pancreatic Ductal Carcinoma.

Figure 1.

Study datasets and design: Dataset curation process for intramural and multi-institutional public CT datasets.

Figure 2.

Convolutional neural network (CNN) architecture and pipeline for classification of CTs into those with PDA or with non-neoplastic pancreas. Bounding box and segmentation mask help narrow the area-of-analysis for the model to the pancreas and peripancreatic tissue. The entire pancreas together with tumor (in CTs with PDA) was created as a single segmentation mask. The model’s pipeline is fully automated and does not depend upon manual segmentations.

Figure 3.

Comparison of model’s performance on intramural test subset (red) and multi-institutional public datasets (blue). Dark solid lines (red or blue) are the observed performance with corresponding 95% confidence intervals depicted as shaded region. The solid gray line is the performance for random chance. The mean AUROC is 0·97 (0·96–0·98) on intramural test subset and 0·90 (0·85–0·94) on the multi-institutional public datasets (p <0·05).

Figure 4:

Model’s performance graph on simulated population with high pre-test probability of PDA and on the pre-diagnostic cohort. The faint grey curves depict the ROCs of multiple bootstrapped test subsets simulating PDA risk of 1–5% (A) and pre-diagnostic cohort (B). The red curve corresponds to mean AUROC (CI) of 0·97 (0·94–0·99) in A, and 0·91 (0·86–0·94) in B. The dashed black line is the performance for random chance.