An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Abstract

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP's clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Fig. 1. Breast cancer neoadjuvant therapy pathway.

a Pre-NAT phase. Breast cancer (BC) is diagnosed following a tumor screening/diagnosis (mammography and/or ultrasound) and biopsy, subsequent histopathology analysis, and a staging breast MRI. These measures help derive demographic, radiological, clinical and histopathological variables describing the patient’s state at diagnosis. cTNM refers to tumor node metastasis. b Mid-NAT phase. The mid-NAT MRI is performed to assess the response and see if therapy adjustments for unresponsive patients. c Post-NAT phase. Breast MRI is used to assess if patients have achieved pathological complete response (pCR). Subsequently, patients undergo surgery, and a histological examination is performed, assessing the post-NAT pathological tumor(ypT) and lymph node staging (ypN) scores (together ypTN), which is the international standard for post-neoadjuvant therapy pathology reporting.

Fig. 2. Workflow of the study.

a Model development. We developed and evaluated a deep learning system to predict the treatment response of breast cancer patients across neoadjuvant therapy (NAT). The system incorporates deep neural networks trained on Pre-NAT mammogram images and longitudinal MRI scans, along with rhpc information (radiological assessments, histopathological assessments, personal patient records, and clinical data). After data retrieval, iMGrhpc and iMRrhpc were modeled independently, where iMGrhpc is based on Pre-NAT mammogram and rhpc data, while iMRrhpc is based on longitudinal MRIs embedding temporal information and rhpc data. Both models include two modules: one module is for cross-modal knowledge learning that predicts rhpc information using only imaging features, and another module is for response prediction using integrated features of rhpc-based and imaging features. These models were further combined into the Multi-modal Response Prediction (MRP) system. MLP refers to a two-layer multi-layer perceptron with an output dimension of 256. b Datasets. The internal dataset was collected from the Netherlands Cancer Institute and was randomly partitioned into training, validation, and test subsets. For evaluating our system on unseen data, we collected three external datasets from different centers: Duke University (United States; n = 288), Fuzhou Province Hospital (China; n = 85), and I-SPY2 (United States; n = 508). c NAT response assessment of AI model and reader study. We assessed MRP's ability to predict pathological response (pCR vs. non-pCR) at different stages-Pre-NAT (before administration of NAT), Mid-NAT (during therapy), and Post-NAT (prior to surgery)-using standard metrics: AUROC (Area Under Receiver Operating Characteristic Curve) and AUPRC (Area Under Precision-Recall Curve). To compare the performance of MRP with human experts, we conducted a reader study involving six international breast radiologists. The average performance of the readers is indicated with a red “+R" in the plot. d Personalizing management in clinical practice. We simulated two scenarios to assess the system’s ability to personalize treatment: identifying non-pCR patients before NAT in whom toxic treatments may be timely adapted, and identifying pCR patients before surgery for the potential reduction of surgical procedures. Circled C indicates current clinical practice; Circled AI indicates our MRP system suggested strategy.

Fig. 3. Six readers averaged performance with Baseline DL models and MRP on internal test sets across the NAT care.

(Top) ROC curves with 95% CIs in bracket calculated with boot-strapping. (Bottom) PRCs with 95% CIs. From left to right: Pre-NAT(Staging), Mid-NAT, Post-NAT(Pre-surgical). rhpc refers to the model trained by radiological assessments (r), histopathological assessments (h), personal patient records (p), and clinical data (c), detailed definitions can be found in Methods and Fig. 1. iMGrhpc is based on Pre-NAT mammogram and rhpc data, while iMRrhpc is based on single/longitudinal MRI(s) embedding with temporal information and rhpc data. MRP aggregates and optimizes the outputs of iMGrhpc model and iMRrhpc model.

Fig. 4. Attributes contribution.

Comparison of coefficient importance among four deep learning models : rhpc (n = 120), iMGrhpc (n = 120), iMRrhpc (n = 120), and MRP (n = 120), organized by coefficients in descending order. The horizontal bar plot displays normalized coefficients, derived from averaged contributing values among cases and normalized across the included attributes, highlighting comparable trends among these attributes. Positive values stand for favorable attributes for response prediction, and vise versa for negative values. The horizontal line (error bar) represents the standard deviation centered on the corresponding coefficients. Each attribute associated different group (radiological(r), histopathological(h), personal(p), clinical(c)) is viewed in the colors legend in the upper right. iMGrhpc is based on Pre-NAT mammogram and rhpc data, while iMRrhpc is based on pre-NAT MRI and rhpc data. MRP aggregates and optimizes the outputs of iMGrhpc model and iMRrhpc model.

Fig. 5. Results of the DCA support using MRP and readers assessment for making therapy-related decisions across NAT scenarios.

Left and Middle: The percentage of net interventions avoided per 1000 patients with non-pCR findings in Pre-/Mid-NAT (y-axis). The black curve (y = 0) of Pre-/Mid-NAT is a standard therapy-all approach (current clinical choice). Right: The percentage of net interventions avoided per 1000 patients with pCR (ypT0) findings in the Post-NAT (Pre-surgical) phase (y-axis). The black curve (y = 0) of Post-NAT is surgery-all approach (current clinical choice).