Combination of ultrasound-based radiomics and deep learning with clinical data to predict response in breast cancer patients treated with neoadjuvant chemotherapy

Abstract

Objectives: Accurate assessment of NAC efficacy is crucial for determining appropriate surgical strategies and guiding the extent of surgical resection in breast cancer. Therefore, this study aimed to design an integrated predictive model combining ultrasound imaging, deep learning features, and clinical characteristics to predict pCR in breast cancer patients undergoing NAC.

Methods: A retrospective study was conducted, including 643 pathologically confirmed breast cancer patients who underwent NAC between January 2022 to February 2024 from two institutions (Center 1: 372 cases; Center 2: 271 cases). Ultrasound images before and after NAC were collected for each patient. A total of 2,920 radiomics features and 4,096 deep learning features were extracted from the ultrasound images. Multiple machine learning algorithms were employed to model and validate the diagnostic performance of different types of features. Finally, clinical data, radiomics, and deep learning features were integrated to form a fusion model, which was evaluated using receiver operating characteristic (ROC) analysis.

Results: The combined model achieved the highest predictive performance for pathological complete response (pCR) across both cohorts. In the internal validation cohort, it reached an accuracy of 0.892 (95% CI: 0.862-0.912) and an AUC of 0.901 (95% CI: 0.854-0.948). In the external cohort, it maintained strong performance with an accuracy of 0.857 (95% CI: 0.822-0.928) and an AUC of 0.891 (95% CI: 0.848-0.934), significantly outperforming the individual models (DeLong test, p < 0.01).The deep learning model showed solid performance with accuracies of 0.875 and 0.833 in the internal and external cohorts, respectively, and AUCs of 0.870 and 0.874. The radiomics model displayed moderate accuracy and AUC in both cohorts, while the clinical model showed the lowest predictive capability among the models, with accuracy and AUC values around 0.67 in both cohorts.

Conclusions: The combined model, integrating clinical, radiomics, and deep learning features, demonstrated superior predictive accuracy for pCR following neoadjuvant chemotherapy (NAC) in breast cancer patients, outperforming individual models. This integrated approach highlights the value of combining diverse data types to improve prediction, offering a promising tool for guiding NAC response assessment and personalized treatment planning.

Keywords: breast cancer; deep learning; neoadjuvant chemotherapy; radiomics; ultrasound.

Figure 1

Diagram of the experimental inclusion-exclusion criteria.

Figure 2

Flow chart. The flowchart shows the study design.

Figure 3

Decision Curve Analysis (DCA) for Predictive Models. (A) DCA curves of six algorithms in the internal validation cohort; (B) DCA results in the external validation cohort. The y-axis represents the net benefit, and the x-axis denotes the threshold probability.

Figure 4

Calibration Curve Analysis. (A) Calibration curves for six classifiers in the internal validation set; (B) Dashed diagonal line indicates perfect calibration. A curve closer to the diagonal suggests better agreement between predicted probability and actual observed frequency of pCR. The XGBoost and MLP models showed the highest calibration accuracy across both datasets.

Figure 5

ROC curves for six classification models in both cohorts. (A) ROC curves in the internal validation cohort. (B) ROC curves in the external validation cohort.

Figure 6

Grad-CAM Visualization of Deep Learning Model Attention in Pre- and Post-NAC Ultrasound Images. This figure demonstrates the deep learning model’s attention maps using Gradient-weighted Class Activation Mapping (Grad-CAM) on tumor ultrasound images before and after neoadjuvant chemotherapy (NAC). (A1, A2) Pre- and post-NAC ultrasound and Grad-CAM images, respectively, of a 53-year-old patient who did not achieve pCR. The Grad-CAM heatmap (A2) highlights strong peripheral activations, particularly on the upper tumor border.B1, B2 Corresponding post-NAC ultrasound and Grad-CAM images of the same non-pCR patient. The attention remains at the edge but appears more diffuse, indicating persistent residual tumor.C1, C2 Pre-NAC ultrasound and Grad-CAM visualization of a 49-year-old patient who achieved pCR. The heatmap (C2) shows dispersed and weak activations across the tumor, suggesting limited model attention toward aggressive patterns.(D1, D2) Post-NAC ultrasound and Grad-CAM of the same pCR patient. The model’s attention in D2 is minimal and centrally located, aligning with radiologic signs of tumor regression.Dashed lines represent the maximal tumor diameters measured during routine clinical evaluation. Tumor sizes were A = 2.24 cm, B = 2.61 cm (non-pCR case), and C = 1.08 cm, D = 0.68 cm (pCR case), respectively. These measurements further validate model attention correlates with tumor shrinkage patterns.

Figure 7

SHAP Plot. The SHAP values illustrate each feature’s contribution to the prediction outcome, providing insight into feature importance and the model’s interpretability.