A Novel Switching of Artificial Intelligence to Generate Simultaneously Multimodal Images to Assess Inflammation and Predict Outcomes in Ulcerative Colitis-(With Video)

Abstract

Objectives: Virtual Chromoendoscopy (VCE) is pivotal for assessing activity and predicting outcomes in Ulcerative Colitis (UC), though interobserver variability and the need for expertise persist. Artificial intelligence (AI) offers standardized VCE-based assessment. This study introduces a novel AI model to detect and simultaneously generate various endoscopic modalities, enhancing AI-driven inflammation assessment and outcome prediction in UC.

Methods: Endoscopic videos in high-definition white-light, iScan2, iScan3, and NBI from UC patients of the international PICaSSO iScan and NBI cohort (302 and 54 patients, respectively) were used to develop a neural network to identify the acquisition modality of each frame and for inter-modality image switching. 2535 frames from 169 videos of the iScan cohort were switched to different modalities and trained a deep-learning model for inflammation assessment. Subsequently, the model was tested on a subset of the iScan and NBI cohorts (72 and 51 videos, respectively). Performance in predicting endoscopic and histological activity and outcomes was evaluated.

Results: The model efficiently classified and converted images across modalities (92% accuracy). Performance in predicting endoscopic and histological remission was excellent, especially with different modalities combined in both iScan (accuracy 81.3% and 89.6%; AUROC 0.92 and 0.89 by UCEIS and PICaSSO, respectively) and the NBI cohort. Moreover, it showed a remarkable ability in predicting clinical outcomes.

Conclusions: Our multimodal "AI-switching" model innovatively detects and transitions between different endoscopic modalities, refining inflammation assessment and outcome prediction in UC by integrating model-derived images.

Keywords: artificial intelligence; ulcerative colitis; virtual chromoendoscopy.

FIGURE 1

AI switching model detection of endoscopic and histological activity in the iScan testing cohorts. The two‐sample test for equality of proportions with continuity correction was statistically significant (p < 0.05) when comparing HD‐WLE + Iscan2 + Iscan3 + NBI versus Iscan2, (b) Iscan3, (c) HD‐WLE + Iscan2 + Iscan3, (d) NBI, (e) HD‐WLE + NBI. Superscripts on each performance measure indicate the modalities in which a significantly difference is reached compared to the multimodal assessment. AUROC, area under the receiver operating characteristic; HD‐WLE, high‐definition white light endoscopy; MCC, Matthew's correlation coefficient; NBI, narrow band imaging; NHI, Nancy histological index; NPV, negative predictive value; PHRI, PICaSSO Histological Remission Index; PICaSSO, Paddington International virtual ChromoendoScopy Score; PPV, positive predictive value; RHI, Robarts histopathology index; UCEIS, ulcerative colitis endoscopic index score.

FIGURE 2

Kaplan Meier curve describing the pooled time‐to‐event analysis of artificial intelligence (AI) vs. human (endoscopists or pathologists) in predicting a major adverse outcome (MAO) over a 12‐month follow‐up in the iScan testing cohort using endoscopic and histological scores (blue and red lines represent disease remission and activity, respectively). The pooled HR was defined in terms of disease activity at least in one endoscopic/histological score.