Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer

Abstract

The achievement of the pathologic complete response (pCR) has been considered a metric for the success of neoadjuvant chemotherapy (NAC) and a powerful surrogate indicator of the risk of recurrence and long-term survival. This study aimed to develop a multimodal deep learning model that combined clinical information and pretreatment MR images for predicting pCR to NAC in patients with breast cancer. The retrospective study cohort consisted of 536 patients with invasive breast cancer who underwent pre-operative NAC. We developed a deep learning model to fuse high-dimensional MR image features and the clinical information for the pretreatment prediction of pCR to NAC in breast cancer. The proposed deep learning model trained on all datasets as clinical information, T1-weighted subtraction images, and T2-weighted images shows better performance with area under the curve (AUC) of 0.888 as compared to the model using only clinical information (AUC = 0.827, P < 0.05). Our results demonstrate that the multimodal fusion approach using deep learning with both clinical information and MR images achieve higher prediction performance compared to the deep learning model without the fusion approach. Deep learning could integrate pretreatment MR images with clinical information to improve pCR prediction performance.

Figure 1

Receiver operating characteristic curves showing the AUC values of different deep learning models in the validation set. (A) ROC curves for the prediction of pretreatment pCR based on different deep learning models trained on clinical information and MR images or only clinical information pCR classifiers. T1 + T2 + C: subtracted-T1W images, T2W images, and clinical information. T1 + C: T1W subtraction images and clinical information. C: clinical information. (B) ROC curves for prediction of pretreatment pCR based on different deep learning models trained on the dataset in the combinations of MR images. T1 + T2: T1W subtraction images and T2W images. T1: T1W subtraction images. T2: T2W images. T1 (lesion): cropped image of the lesion in T1W subtraction images.

Figure 2

Deep learning architectures for the multimodal pCR prediction model. (A) The feature extractors for contrast-enhanced T1W subtraction MR images and T2W MR images were used in two 3D ResNet-50. The MR images for the input were subjected to isotropic transformation and cropped to a 3D form of 224 × 224 × 64. (B) FC layer was used for clinical information inputs. The outputs of each 3D ResNet-50 and FC layer for clinical information were concatenated. The final FC layer with sigmoid activation function was used in the prediction of pCR.