Development of deep learning-based narrow-band imaging endocytoscopic classification for predicting colorectal lesions from a retrospective study

Abstract

Data-driven approaches have advanced colorectal lesion diagnosis in digestive endoscopy, yet their application in endocytoscopy (EC)-a high-magnification imaging technique-remains limited, with most studies relying on conventional machine learning methods like support vector machines. Inspired by the success of large-scale language models that leverage progressive pre-training, we develop a computer-aided diagnosis (CAD) model using narrow-band imaging endocytoscopy (EC-NBI) to classify colorectal lesions (non-neoplastic lesions, adenomas, and invasive cancers). Here, we show that our model, trained through a multi-stage pre-training strategy combined with supervised deep clustering, outperforms state-of-the-art supervised methods in a multi-center retrospective cohort. Notably, it surpasses endoscopists' diagnostic accuracy in human-machine competitions and enhances their performance when used as an assistive tool. This EC-NBI CAD model significantly improves the accuracy and consistency of diagnosing colorectal lesions, laying a foundation for future early cancer screening, particularly for distinguishing superficial and deep submucosal invasive cancers, pending further expansive multi-center data.

Fig. 1

Flowcharts of the dataset.

Fig. 2. Research flowchart.

a Study design and the flowchart of the multi-stage continuing pre-training (CPT) strategy. b The model evaluation via different metrics.

Fig. 3. The performance indicators of the model at different levels in both the independent internal and external validation cohort.

At the image level, we quantitatively perceive the model performance by calculating the receiver operating characteristic (ROC) (a) and confusion matrix (b) of the training and validation cohorts of the model. Further, through the T-SNE method, we perform dimensionality reduction and visualization of the features of the samples. The clustering situation between different classes can be observed (c), and their silhouette scores are quantitatively calculated. The same indicators calculation and visualization are also displayed at the lesion level (d–f).

Fig. 4. Comparison experiments of different methods.

a At the image level, the performance of different comparison methods on different disease types in the training cohort. b At the image level, the performance of different comparison methods on different disease types in the internal validation cohort. c At the image level, the performance of different comparison methods on different disease types in the external validation cohort. d At the lesion level, the performance of different comparison methods on different disease types in the training cohort. e At the image level, the performance of different comparison methods on different disease types in the internal validation cohort. f At the image level, the performance of different comparison methods on different disease types in the external validation cohort. Note: The closer to the periphery, the better the performance. The numerical comparison below each radar chart represents the area enclosed by the five indicators under the different methods; the larger the area, the higher the comprehensive performance of the method.

Fig. 5. Human-machine collaboration.

Performance comparison between our model and endoscopists, as well as the improvement in endoscopists’ prediction results with the assistance of AI. (AI) means this doctor has AI assistance. The solid dark-red curves represent AI-assisted endoscopist diagnoses. The dashed light-red curves represent endoscopist-only diagnoses. The solid blue curves represent AI-only diagnoses (Our method).

Fig. 6. Model interpretability.

a The class activation mapping of the model when predicting samples of different categories was obtained through Gradient-weighted Class Activation Mapping (Grad-CAM). b Bee swarm summary plot of feature importance based on Shapley Additive exPlanations (SHAP) analysis. The bee swarm plot is designed to display an information-dense summary illustrating how the top features in a dataset affect the output of a model. Each observation in the data is represented by a single dot on each feature row. The vertical axis represents the features, sorted from top to bottom according to their importance as predictors. The position of a dot on a feature row is determined by the SHAP value of the corresponding feature, and the accumulation of dots on each feature row illustrates its density. The feature value determines the color of the dots, with red indicating large SHAP values and blue indicating small SHAP values. c Feature importance plot. Passing the SHAP value matrix to the bar plot function creates a global feature importance plot for each class, where the global importance of each feature for each class is considered as the average absolute value of that feature overall given samples.