Interpretable multi-modal artificial intelligence model for predicting gastric cancer response to neoadjuvant chemotherapy

Abstract

Neoadjuvant chemotherapy assessment is imperative for prognostication and clinical management of locally advanced gastric cancer. We propose an incremental supervised contrastive learning model (iSCLM), an interpretable artificial intelligence framework integrating pretreatment CT scans and H&E-stained biopsy images, for improved decision-making regarding neoadjuvant chemotherapy. We have constructed and tested iSCLM using retrospective data from 2,387 patients across 10 medical centers and evaluated its discriminative ability in a prospective cohort (132 patients; ChiCTR2300068917). iSCLM achieves areas under receiver operating characteristic curves of 0.846-0.876 across different test cohorts. Computed tomography (CT) and pathological attention heatmaps from Shapley additive explanations and global sort pooling illustrate additional benefits for capturing morphological features through supervised contrastive learning. Specifically, pathological top-ranked tiles exhibit decreased distances to tumor-invasive borders and increased inflammatory cell infiltration in responders compared with non-responders. Moreover, CD11c expression is elevated in responders. The developed interpretable model at the molecular pathology level accurately predicts chemotherapy efficacy.

Keywords: artificial intelligence; computed tomography; gastric cancer; neoadjuvant chemotherapy; whole-slide image.

Graphical abstract

Figure 1

Workflow of the study (A) Feature extraction and model development. L_con, the supervised contrastive learning loss; L_cel, the cross-entropy loss. (B) External validation and patient risk stratification. (C) Biological interpretation and molecular verification of the model. iRM, incremental radiology model; iPM, incremental pathology model; iSCLM, incremental supervised contrastive learning model.

Figure 2

Prediction performance in the external and prospective test cohorts Receiver operating characteristic curve (ROC) of iSCLM, iRM, iRPM and iPM for predicting tumor response to neoadjuvant chemotherapy. (A) External test cohort 1, (B) external test cohort 2, (C) prospective test cohort. iSCLM, incremental supervised contrastive learning model; iRM, incremental radiology model; iRPM, model constructed by direct concatenation of radiological and pathological features with incremental learning; iPM, incremental pathology model.

Figure 3

Spatial analysis of model attention and cell component analysis (A) Original whole-slide image (WSI), manually annotated tumor-invasive border, tiles rank heatmap (left), and distribution of top-ranked tiles (right). (B) Manual annotation (left), cell classification generated by HoVer-Net (middle), and pie chart of cell classification in each tile (right) for two typical patients, one responder and one non-responder. Examples of the cell components in the tumor region and invasive border are demonstrated (middle). (C) Inflam/NEP ratio of the top-10 tiles (left) and the whole tiles (right) in responders and non-responders. The boxplot summarizes data using the median and interquartile range to display its distribution and variability. Inflam, inflammatory cells; NEP, neoplastic epithelial cells.

Figure 4

RNA-seq and IHC results (A) Heatmap plot presenting cell composition based on ssGSEA of RNA-seq. (B) Heatmap plot presenting the activity of pathways identified using RNA-seq. The colors in the heatmap correspond to the activity scores of each pathway, as quantified through gene set variation analysis. (C) Heatmap plot depicting the expression levels of inflammatory cell-related biomarkers derived from RNA-seq. (D) Raincloud plot illustrating the differential enrichment of dendritic cells between responders and non-responders as analyzed using ssGSEA of RNA-seq. (E) Raincloud plot displaying the variation in CD11c expression between responders and non-responders, based on RNA-seq data. (F) Heatmap plot illustrating variations in cell marker expression between responders and non-responders, evaluated using IHC. (G) Raincloud plot indicating the proportion of CD11c-positive cells in responders versus non-responders, as determined using IHC. (H) Raincloud plot illustrating the proportion of CD163-positive cells in responders and non-responders based on IHC. The raincloud plot combines an illustration of data distribution (the “cloud”), with jittered raw data (the “rain”). The boxplot summarizes data using the median and interquartile range to display its distribution and variability. (I) Multiplex immunohistochemistry (mIHC) staining for CD4 (purple), CD8 (orange), CD163 (yellow), CD11c (green), and DAPI (blue) of a responder example. (J) mIHC staining for CD4 (purple), CD8 (orange), CD163 (yellow), CD11c (green), and DAPI (blue) of a non-responder example. RNA-seq, RNA sequencing; IHC, immunohistochemistry; ssGSEA, single-sample gene set enrichment analysis; DC, dendritic cell.

Figure 5

Examples of correction attributed to changes in the focus areas of CT imaging and pathology in the prospective cohort (A) A responder, with a true label of 1. iRM incorrectly provides a predicted value output of 0.300, while iSCLM corrects this to 0.993 after being reinforced by pathological feature representation. The CT attention map of representative images generated by Shapley additive explanations, which presents the last layer of Convn_x of ResNet-34 and helps explain this correction. IHC shows this patient has a high CD11c expression. (B) A responder, with a true label of 1. iPM incorrectly provides a predicted value output of 0.480, whereas iSCLM predicts 0.999, reinforced by CT feature representation. The attention heatmap of iPM generated by global sort pooling helps explain this correction. The optimal threshold for prediction in different cohorts was determined using the Youden index in the development cohort. iSCLM, incremental supervised contrastive learning model; iRM, incremental radiology model; iPM, incremental pathology model; CT, computed tomography; WSI, whole-slide images.