A multimodal and fully automated system for prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer

Abstract

Accurately predicting pathological complete response (pCR) before neoadjuvant chemotherapy (NAC) is crucial for patients with breast cancer. In this study, we developed a multimodal integrated fully automated pipeline system (MIFAPS) in forecasting pCR to NAC, using a multicenter and prospective dataset of 1004 patients with locally advanced breast cancer, incorporating pretreatment magnetic resonance imaging, whole slide image, and clinical risk factors. The results demonstrated that MIFAPS offered a favorable predictive performance in both the pooled external test set [area under the curve (AUC) = 0.882] and the prospective test set (AUC = 0.909). In addition, MIFAPS significantly outperformed single-modality models (P < 0.05). Furthermore, the high deep learning scores were associated with immune-related pathways and the promotion of antitumor cells in the microenvironment during biological basis exploration. Overall, our study demonstrates a promising approach for improving the prediction of pCR to NAC in patients with breast cancer through the integration of multimodal data.

Fig. 1.. Patient enrollment workflow.

MRI, magnetic resonance imaging.

Fig. 2.. Study workflow.

(A) MRI-based deep learning signature development. (B) WSI-based deep learning signature development. (C) Clinical signature development. (D) MIFAPS integrated clinical signature, pretreatment MRI-based deep learning signature, and WSI-based deep learning signature. (E) The biological basis of deep learning was explored using RNA sequencing data from 67 patients. (F) The predictive efficiency of MIFAPS was assessed in the internal/external/prospective test sets, and its underlying predictive mechanism was investigated with visual analysis. (G) Potential clinical impact of the MIFAPS. WSI, whole slide image; MRI, magnetic resonance imaging; MIFAPS, multimodal integrated fully automated pipeline system; pCR, pathological complete response; ROC, receiver operating characteristics.

Fig. 3.. Performance of different models for predicting the pCR to NAC.

ROC curves of different models in the training and validation sets (A), internal test set (B), pooled external test set (C), and prospective test set (D). PR curves of different models in the training and validation sets (E), internal test set (F), pooled external test set (G), and prospective test set (H). ROC curves of MIFAPS in different molecular subtypes on the pooled external test set (I) and prospective test set (J) and different lesion size subgroups on the pooled external test set (K) and prospective test set (L). Scatter graphs describing the MIFAPS correct cases falsely predicted by the MRI (M), WSI (N), and clinical (O) models in the pooled external test set. Scatter graphs describing the MIFAPS correct cases falsely predicted by the MRI (P), WSI (Q), and clinical (R) models in the prospective test set. pCR, pathological complete response; ROC, receiver operating characteristic; AUC, area under the ROC curve; WSI, whole slide image; MRI, magnetic resonance imaging; MIFAPS, multimodal integrated fully automated pipeline system; HER2, human epidermal growth factor receptor 2.

Fig. 4.. MIFAPS-assisted clinical decision-making.

Confusion matrix of the MIFAPS in the pooled external test set (A) and prospective test set (B). Recommendation for therapies according to the MIFAPS in the pooled external test set (C) and prospective test set (D).

Fig. 5.. Multimodal model visualization.

(A) Heatmaps obtained based on Grad-CAM from the MRI. The red and yellow regions have higher predictive significance than the green and blue regions. (B) Histopathology WSIs: the original pathological images of breast cancer in patients with high scores (top) and low scores (bottom). (C) The attention heatmap visualizes the heatmap used to assign different attention scores. The red represents areas of high attention, whereas the blue represents areas of low attention. (D) High-attention tiles. (E) Segmented and classified cell types of the highest attention tiles. (F) Quantification of cell types in high-attention tiles. * indicates significant difference; n.s., not significant; WSI, whole slide image; MRI, magnetic resonance imaging.

Fig. 6.. Biologic basis of the MIFAPS.

(A) Volcano diagram of gene expression profiles in samples separated by low and high scores. (B) Top 5 up-regulated pathways in high scores via GSEA based on the GO database. (C) Box plot representing the estimation of the abundances of member cell types in a mixed cell population. (D) IHC staining of CD4 memory T cells, M1 macrophage, and M2 macrophage from a breast cancer patient with a high score and another one with a low score. A 51-year-old woman with breast cancer with a high score (left). A 70-year-old woman with breast cancer with a low score (right). CD4 memory T cells and M2 macrophages appear brown. M1 macrophages appear red and green. MIFAPS, multimodal integrated fully automated pipeline system; GSEA, gene set enrichment analysis; GO, Gene Ontology.

Fig. 7.. Misclassified cases analysis.

(A) Heatmap visualization of the MRI-based deep learning model. (a) The pretreatment image of a 50-year-old woman with non-pCR to NAC in breast cancer was misclassified as pCR. The MRI shows a 5.8-cm lesion. (b) The pretreatment image of a 30-year-old woman with pCR to NAC in breast cancer was misclassified as non-pCR. The MRI shows a 2.3-cm lesion. (B) Segmented and classified cell types of the tile with the highest attention in WSI. Images (c) and (a) belong to the same patient. Images (d) and (b) belong to the same patient. (C) Quantification of cell types in high-attention tiles in pCR and non-pCR WSIs. (D) Top 5 enriched pathways via GSEA based on the GO database. * indicates significant difference; WSI, whole slide image; MRI, magnetic resonance imaging; pCR, pathological complete response; NAC, neoadjuvant chemotherapy; GSEA, gene set enrichment analysis; GO, Gene Ontology.