Use of Large Language Models to Determine the Surveillance Colonoscopy Interval: A Bi-institutional Validation Study

Abstract

Introduction: To determine the appropriate post-polypectomy colonoscopy surveillance interval, endoscopists synthesize information from colonoscopy and pathology report impressions and subsequently apply guideline-recommended interval algorithms, such as those developed by the United States Multi-Society Task Force (USMSTF). Given the complexity of these guidelines, this manual process is error-prone, necessitating automated tools, including large language models (LLMs), to improve guideline adherence.

Objective: The primary aim of this study was to identify the LLM performance in determining the guideline-concordant post-polypectomy surveillance interval on a cohort of 1000 real-world colonoscopy and pathology report impressions.

Methods: The data of patients who underwent a screening or surveillance colonoscopy in 2023-2024 at two academic health centers were included. Using a custom prompt outlining the USMSTF post-polypectomy surveillance algorithm, the LLM (GPT-4o) was asked to determine the appropriate surveillance interval for all 1000 examples in the dataset. This experiment, using the same model, prompt, and dataset, was repeated 10 times; all experiments were conducted between January 27, 2025, and February 3, 2025.

Results: Across 10 experiments, the average accuracy was 94.6%. There was no significant difference in accuracy based on the institution from which the data originated or the presence of synchronous upper GI endoscopy data within the pathology report impression. Examples with 1-3 colon polyps had an average accuracy of 95.8% while examples with 4 or more colon polyps had an average accuracy of 88.2%, combined p-value < 0.001.

Conclusion: LLMs with a custom prompt achieve consistently high accuracy in determining the guideline-based surveillance colonoscopy interval.