Deep learning model with pathological knowledge for detection of colorectal neuroendocrine tumor

Abstract

Colorectal neuroendocrine tumors (NETs) differ significantly from colorectal carcinoma (CRC) in terms of treatment strategy and prognosis, necessitating a cost-effective approach for accurate discrimination. Here, we propose an approach for distinguishing between colorectal NET and CRC based on pathological images by utilizing pathological prior information to facilitate the generation of robust slide-level features. By calculating the similarity between morphological descriptions and patches, our approach selects only 2% of the diagnostically relevant patches for both training and inference, achieving an area under the receiver operating characteristic curve (AUROC) of 0.9974 on the internal dataset, and AUROCs of 0.9724 and 0.9513 on two external datasets. Our model effectively identifies NETs from CRCs, reducing unnecessary immunohistochemical tests and enhancing the precise treatment for patients with colorectal tumors. Our approach also enables researchers to investigate methods with high accuracy and low computational complexity, thereby advancing the application of artificial intelligence in clinical settings.

Keywords: colorectal cancer; colorectal neuroendocrine tumor; deep learning; foundation model; multimodal fusion; pathological knowledge.

Graphical abstract

Figure 1

The workflow of the proposed deep learning model (A) The datasets were collected from three different centers containing more than 1,500 patients. Data from one center were used as an internal dataset for model training, while data from the other two centers were utilized to construct external datasets for testing the model’s generalization. (B) The image encoder and the text encoder employed in our model were trained through contrastive learning on large-scale pathology image-text pairs. This training strategy enhanced the encoders’ capabilities, enabling them to capture more robust representations. (C) After digitizing the slides, the tissue regions were segmented, and the whole-slide images were decomposed into patches. (D) A similarity-based selection method was used to extract diagnostically relevant patches from the whole-slide images. (E) The computational flow of the model is mainly divided into three parts: diagnosis-related patch selection, text-guided slide-level feature generation, and prediction. (F) Our model can be applied in clinical settings for early screening, significantly reducing the workload of pathologists and minimizing the need for additional diagnostic testing.

Figure 2

The performance of our models on three datasets: an internal dataset (Internal-FAH) and two external datasets (External-CC and External-TCIH) (A–C) The performance of the model with varying numbers of selected patches. (A), (B), and (c) correspond to the Internal-FAH, External-CC, and External-TCIH datasets, respectively. (D) The model achieved an AUROC score of 0.9974 on the internal dataset. (E) On the External-CC dataset, the model maintained an AUROC of 0.9724. (F) On the External-TCIH dataset, the model achieved an AUROC of 0.9513. (G) When compared to existing models on the external datasets, our model consistently outperformed them.

Figure 3

Visualization of selected patches (A–C) The left figure displays the manual outlines performed by the pathologist. The middle figure shows patches selected in different proportions projected onto the image. The right figure presents representative patches chosen based on similarity. Correct proportion indicates the proportions of model-selected patches within the pathologist’s labeled region.

Figure 4

Visualization results using UMAP (A–C) Results were obtained from Internal-FAH, External-CC, and External-TCIH datasets, respectively. The left-side figures show results after random sampling from all patches, while the right-side figures display results after random sampling from patches selected using the proposed method. The features selected by the proposed method demonstrated greater separability, which enhanced model training.