Artificial intelligence and radiologists in pancreatic cancer detection using standard of care CT scans (PANORAMA): an international, paired, non-inferiority, confirmatory, observational study

Abstract

Background: Pancreatic ductal adenocarcinoma (PDAC) has the worst prognosis among major cancer types, primarily due to late diagnosis on contrast-enhanced CT. Artificial intelligence (AI) can improve diagnostic performance, but robust benchmarks and reliable comparison to radiologists' performance are scarce. We established an open-source benchmark with the aim of investigating AI systems for PDAC detection on CT and compared them to radiologists' performance, at scale.

Methods: In this international, paired, non-inferiority, confirmatory, observational study (PANORAMA), the AI system was trained and externally validated within an international benchmark, with a cohort of 2310 patients from four tertiary care centres in the Netherlands and the USA for training (n=2224) and tuning (n=86), and a sequestered cohort of 1130 patients from five tertiary care centres (the Netherlands, Sweden, and Norway) for testing. A multi-reader, multi-case observer study with 68 radiologists (40 centres, 12 countries; median 9·0 [IQR 6·0-14·5] years of experience) was conducted on a subset of 391 patients from the testing cohort. The reference standard was established with histopathology and at least 3 years of clinical follow-up. The primary endpoint was the mean area under the receiver operating characteristic curve (AUROC) of the AI system compared to that of radiologists at PDAC detection on CT. The study protocol and statistical plan were prespecified to test non-inferiority (considering a margin of 0·05), followed by superiority towards the AI system. This study is registered with Zenodo (https://doi.org/10.5281/zenodo.10599559) and is complete.

Findings: Of the 3440 (1511 [44%] female, 1929 [56%] male; median age 67 [IQR 58-74] years) included patients (Jan 1, 2004 to Dec 31, 2023), 1103 (32%) received a positive PDAC diagnosis. In the sequestered testing cohort of 1130 patients (406 with histologically confirmed PDAC), AI achieved an AUROC of 0·92 (95% CI 0·90-0·93). In the subset of 391 patients (144 [37%] with histologically confirmed PDAC) used for the reader study, AI achieved statistically non-inferior (p<0·0001) and superior (p=0·001) performance with an AUROC of 0·92 (95% CI 0·89-0·94), compared to the pool of 68 participating radiologists, with an AUROC of 0·88 (0·85-0·91).

Interpretation: AI demonstrated substantially improved PDAC detection on routine CT scans compared to radiologists on average, showing potential to detect cancer earlier and improve patient outcomes.

Funding: European Union's Horizon 2020 research and innovation programme.