Enhancing colorectal polyp classification using gaze-based attention networks

Abstract

Colorectal polyps are potential precursor lesions of colorectal cancer. Accurate classification of colorectal polyps during endoscopy is crucial for early diagnosis and effective treatment. Automatic and accurate classification of colorectal polyps based on convolutional neural networks (CNNs) during endoscopy is vital for assisting endoscopists in diagnosis and treatment. However, this task remains challenging due to difficulties in the data acquisition and annotation processes, the poor interpretability of the data output, and the lack of widespread acceptance of the CNN models by clinicians. This study proposes an innovative approach that utilizes gaze attention information from endoscopists as an auxiliary supervisory signal to train a CNN-based model for the classification of colorectal polyps. Gaze information from the reading of endoscopic images was first recorded through an eye-tracker. Then, the gaze information was processed and applied to supervise the CNN model's attention via an attention consistency module. Comprehensive experiments were conducted on a dataset that contained three types of colorectal polyps. The results showed that EfficientNet_b1 with supervised gaze information achieved an overall test accuracy of 86.96%, a precision of 87.92%, a recall of 88.41%, an F1 score of 88.16%, the area under the receiver operating characteristic (ROC) curve (AUC) is 0.9022. All evaluation metrics surpassed those of EfficientNet_b1 without gaze information supervision. The class activation maps generated by the proposed network also indicate that the endoscopist's gaze-attention information, as auxiliary prior knowledge, increases the accuracy of colorectal polyp classification, offering a new solution to the field of medical image analysis.

Keywords: Class activation map; Colorectal polyps; Eye-tracking; Gaze attention.

Figure 1. Illustration of different types of colorectal polyps and corresponding gaze attention heatmaps.

(A) Diagram of a hyperplastic or inflammatory polyp; (B) gaze attention heatmaps of a hyperplastic or inflammatory polyp; (C) diagram of an adenomatous polyp, intramucosal carcinoma, and superficial submucosal invasive carcinoma; (D) gaze attention heatmaps of an adenomatous polyp, intramucosal carcinoma, and superficial submucosal invasive carcinoma; (E) diagram of a deep submucosal invasive carcinoma; (F) gaze attention heatmaps of a deep submucosal invasive carcinoma.

Figure 2. Architecture of EfficientNet_b1 backbone.

Figure 3. Architecture of gaze-based attention network for colorectal polyp classification.

Figure 4. Confusion matrices of EfficientNet_b1.

(A) Confusion matrix without gaze attention enabled; (B) confusion matrix with gaze attention enabled.

Figure 5. ROC curves of EfficientNet_b1 and EfficientNet_b1+Gaze.

Figure 6. Demonstration of the effects of enabling or disabling gaze attention.

The first category shows pictures of the test; the second column shows the effect of using CAM; and the third column shows the effect of adding gaze attention.