Exploring personalized neoadjuvant therapy selection strategies in breast cancer: an explainable multi-modal response model

Abstract

Background: Neoadjuvant therapy (NAT) regimens for breast cancer are generally determined according to cancer stage and molecular subtypes without fully considering the inter-patient variability, which may lead to inefficiency or overtreatment. Artificial intelligence (AI) may support personalized regimen recommendations by learning the synergistic relationship between pre-NAT individual-patient data, regimens, and corresponding short- or long-term therapy responses.

Methods: In this retrospective study, we collected data from breast cancer patients treated with NAT between 2000 and 2020 from the Netherlands and the USA. Median follow-up times ranged from 3·7 to 4·9 years across molecular subtypes and cohorts. We developed and externally validated a multi-modal model integrating pre-NAT clinical data, dynamic contrast enhanced (DCE)-MRI images, and medical reports to predict pathological complete response (pCR) and likelihood of survival after NAT. We subsequently evaluated potential benefits for patients receiving a personalized regimen recommended based on these predictions.

Findings: We trained our model on 655 patients and validated it on internal (655 patients) and external (241 patients) cohorts. Given the factual regimens, the model can correctly predict the corresponding therapy response, with areas under the receiver operating characteristic curves (AUC) of 0·80 (95% CI 0·73-0·87), 0·75 (0·66-0·83), and 0·85 (0·77-0·92) for pCR prediction of human epidermal growth factor receptor 2 (HER2)+, triple-negative, and estrogen receptor/progesterone receptor (ER/PR)+&HER2- patients in the internal validation cohort, respectively. Performance in the external validation cohort was 0·707 (0·557-0·836), 0·558 (0·359-0·749), and 0·860 (0·767-0·945) for the corresponding molecular subtypes, respectively. In the internal validation cohort, survival prediction identified high-risk patients across different molecular subtypes, as demonstrated by a hazard ratio (HR) of 3·29 (0·91-11·94) (HER2+), 3·54 (1·52-8·20) (triple-negative), and 2·78 (1·45-5·31) (ER/PR+&HER2-), albeit results were not significant for HER2+ cancers.

Interpretation: Our findings indicate that the prognostic scores generated by the response model could identify patient subgroups with relatively poor outcomes under their actual treatments. These preliminary findings may inform future efforts toward personalized NAT regimen selection beyond traditional criteria such as cancer stage and subtype, but should be interpreted cautiously and validated in prospective studies with longer follow-up because these tumors can relapse at a later stage.

Funding: None.

Keywords: Breast cancer; Explainable artificial intelligence; Multi-modal learning; Neoadjuvant therapy; Precise medicine.

Fig. 1

Clinical need and outline of the study. (A) Guideline-based neoadjuvant therapy (NAT) regimen selection and prognostic outcomes for breast cancer patients in clinical practice. (B) Combined with imaging and report data, the deep learning model can offer personalized NAT regimen recommendations and improve patient prognosis. (C) Flowchart of personalized NAT regimen recommendation based on outcome scores predicted from the model. (D) Various prognostic-related posterior distributions were derived from multi-modal data using multiple encoders. These distributions were then combined into a single distribution using the Product-of-Experts (PoE) approach. The combined distribution and potential NAT regimens were utilized to predict prognosis outcomes, including pathological complete response (pCR) and risk scores. IHC: immunohistochemistry. SISH: silver-enhanced in situ hybridization.

Fig. 2

Prognosis performance of pathological complete response (pCR) scores and risk scores predicted by response models in the NKI internal validation set and DUKE external validation cohort. ROC curves for pCR status prediction are presented for HER2+, triple-negative, and ER/PR+&HER2− in NKI validation set (A) and DUKE cohort (C). For survival analysis, patients were stratified by predicted risk score as high risk or low risk. The 25% cutoff of the predicted risk score of the training set was used as the cutoff (−0·394, 0·424, 0·570). Kaplan–Meier curves for overall survival are presented for HER2+, triple-negative, and ER/PR+&HER2− in NKI validation set (B) and DUKE cohort (D). The p-values in the ROC analysis were derived using t-tests comparing each method against our proposed approach.

Fig. 3

Visualization of model interpretability in the NKI internal validation set. The permutation importances for each clinical variable in predicting pathological complete response (pCR) (A) and survival (B) are listed. Data were shown as mean ± standard deviation. The gradient-weighted class activation mapping (Grad-CAM) heatmaps of the dynamic contrast enhanced (DCE)-MRI are displayed in (C) in the following order: pre-contrast DCE-MRI at the top left, post-contrast DCE-MRI at the top right, segmentation mask at the bottom left, and heatmap at the bottom right. The correlation heatmaps of the reports are displayed in (D) in the following order from top to bottom: radiology report, pathology report, and medical record for each patient. The density histograms of the prognosis distributions are presented in (E). $\Delta \mu$ indicates the correction of the mean between Product-of-Experts (PoE) and clinical data.

Fig. 4

Benefits of artificial intelligence (AI) recommendations in different factual regimen subgroups of patients in the NKI internal validation set. Our AI recommendation system can recommend lower/higher-toxicity regimens or clinical trials for different patient populations (A). Decision curve for AI recommendation on patients with factual regimens of lower-toxicity regimens (B) and patients with factual regimens of higher-toxicity regimens (D). Kaplan–Meier survival curve for patients applying model-based three-category regimen recommendation on patients with factual regimens of lower-toxicity regimens (C) and patients with factual regimens of higher-toxicity regimens (E). The p-values in the Kaplan–Meier survival curve analysis were derived using log-rank tests comparing each regimen against the reference regimen.