Effects of artificial intelligence assistance on endoscopist performance: Comparison of diagnostic performance in superficial esophageal squamous cell carcinoma detection using video-based models

Abstract

Objectives: Superficial esophageal squamous cell carcinoma (ESCC) detection is crucial. Although narrow-band imaging improves detection, its effectiveness is diminished by inexperienced endoscopists. The effects of artificial intelligence (AI) assistance on ESCC detection by endoscopists remain unclear. Therefore, this study aimed to develop and validate an AI model for ESCC detection using endoscopic video analysis and evaluate diagnostic improvements.

Methods: Endoscopic videos with and without ESCC lesions were collected from May 2020 to January 2022. The AI model trained on annotated videos and 18 endoscopists (eight experts, 10 non-experts) evaluated their diagnostic performance. After 4 weeks, the endoscopists re-evaluated the test data with AI assistance. Sensitivity, specificity, and accuracy were compared between endoscopists with and without AI assistance.

Results: Training data comprised 280 cases (140 with and 140 without lesions), and test data, 115 cases (52 with and 63 without lesions). In the test data, the median lesion size was 14.5 mm (range: 1-100 mm), with pathological depths ranging from high-grade intraepithelial to submucosal neoplasia. The model's sensitivity, specificity, and accuracy were 76.0%, 79.4%, and 77.2%, respectively. With AI assistance, endoscopist sensitivity (57.4% vs. 66.5%) and accuracy (68.6% vs. 75.9%) improved significantly, while specificity increased slightly (87.0% vs. 91.6%). Experts demonstrated substantial improvements in sensitivity (59.1% vs. 70.0%) and accuracy (72.1% vs. 79.3%). Non-expert accuracy increased significantly (65.8% vs. 73.3%), with slight improvements in sensitivity (56.1% vs. 63.7%) and specificity (81.9% vs. 89.2%).

Conclusions: AI assistance enhances ESCC detection and improves endoscopists' diagnostic performance, regardless of experience.

Keywords: artificial intelligence; esophageal neoplasms; esophageal squamous cell carcinoma; gastrointestinal endoscopy; narrow band imaging.

FIGURE 1

Flowchart detailing the procedures employed in developing and evaluating the AI model for detecting superficial esophageal squamous cell carcinoma, including the assessment of its assistive effects on the performance of endoscopists. AI, artificial intelligence; NCCHE, National Cancer Center Hospital East.

FIGURE 2

Overview of the data preparation and annotation process for lesion frames. 1. Videos were divided into individual frames of 30 fps each. 2. Lesion frames were annotated using the Visual Object Tagging Tool to create a ground‐truth dataset.

FIGURE 3

Definition of lesion detection using the AI model. 1. Per‐frame analysis involved evaluating the IoU with a threshold of ≥ 0.3 and comparing the ground truth and AI predictions for each frame. The IoU was calculated as the area of overlap divided by the area of union. 2. Per‐lesion analysis considered detection over five consecutive frames. AI, artificial intelligence; IoU, intersection over union.

FIGURE 4

Diagnostic performance of all endoscopists with and without AI assistance. Significant improvements were observed with AI support across all evaluation criteria; particularly, a substantial enhancement in sensitivity and accuracy was noted. AI: artificial intelligence, PPV: positive predictive value, NPV: negative predictive value.

FIGURE 5

Diagnostic performance of expert endoscopists with and without AI assistance. Although there was a marginal improvement in the initially high sensitivity and PPV, statistical significance was not achieved. Notably, substantial enhancements were observed in sensitivity and accuracy. AI, artificial intelligence; PPV, positive predictive value; NPV, negative predictive value.

FIGURE 6

Diagnostic performance of non‐expert endoscopists with and without AI assistance. Significant improvements with AI support were observed across all evaluation criteria. A substantial improvement in specificity, sensitivity, and accuracy was a distinctive finding. AI, artificial intelligence; PPV, positive predictive value; NPV, negative predictive value.

FIGURE 7

Examples of video‐captured images diagnosed using the AI model Images (a) and (d) depict instances of diagnosis without AI support, whereas images (b) and (e) represent cases diagnosed with AI assistance. The AI model accurately identified subtle changes in lesions in both distant and close‐up views. Images (c) and (f) served as the reference iodine‐stained images used in the creation of the ground truth. AI, artificial intelligence.