Multimodal Spatiotemporal Deep Learning Framework to Predict Response of Breast Cancer to Neoadjuvant Systemic Therapy

Abstract

Current approaches to breast cancer therapy include neoadjuvant systemic therapy (NST). The efficacy of NST is measured by pathologic complete response (pCR). A patient who attains pCR has significantly enhanced disease-free survival progress. The accurate prediction of pCR in response to a given treatment regimen could increase the likelihood of achieving pCR and prevent toxicities caused by treatments that are not effective. Th early prediction of response to NST can increase the likelihood of survival and help with decisions regarding breast-conserving surgery. An automated NST prediction framework that is able to precisely predict which patient undergoing NST will achieve a pathological complete response (pCR) at an early stage of treatment is needed. Here, we propose an end-to-end efficient multimodal spatiotemporal deep learning framework (deep-NST) framework to predict the outcome of NST prior or at an early stage of treatment. The deep-NST model incorporates imaging data captured at different timestamps of NST regimens, a tumor's molecular data, and a patient's demographic data. The efficacy of the proposed work is validated on the publicly available ISPY-1 dataset, in terms of accuracy, area under the curve (AUC), and computational complexity. In addition, seven ablation experiments were carried out to evaluate the impact of each design module in the proposed work. The experimental results show that the proposed framework performs significantly better than other recent methods.

Keywords: 3D-CNN multimodal framework; automated neoadjuvant systematic therapy prediction; multimodal deep learning framework.

Figure 1

The proposed deep-NST prediction framework architecture. Image sequences with different time stamps (represented by $T_1$, $T_2$, $T_3$, and $T_4$) are processed through parallel 3D-CNN networks. The feature outputs from all time stamps and clinical reports are added and processed by a final soft-max layer. Further trained weights of the first 3D-CNN network are fine-tuned over image sequences captured at the early stage of the treatment ($T_1$) along with the clinical features. The numbers in the image denote the size of the feature dimension at each layer.

Figure 2

Data sample distribution in pCR0 and pCR1 classes of ISPY-1. In the original dataset, each patient was represented by a single instance of MRI scans. However, in the balanced dataset, we have made updates to include multiple MRI scans at the same time-stamp for patients who responded to the NST.

Figure 3

ROC curves for pCR prediction models on the ISPY-1 dataset: (a) conventional 3D-VGGNet (AUC = 0.68), (b) conventional 3D-ResNet (AUC = 0.50), and (c) proposed deep-NST (AUC = 0.88).

Figure 4

Confusion matrices of (a) 3D-VGGNet, (b) 3D-ResNet, and (c) proposed deep-NST frameworks.

Figure 5

ROC curves for the proposed model on different inputs: (a) only $T_1$ MRI scans; (b) $T_1$ MRI scans with clinical information; (c) $T_1$ and $T_2$ MRI scans with clinical information; and (d) $T_1$, $T_2$, and $T_3$ MRI scans with clinical information, over ISPY-1 dataset.

Figure 6

Confusion matrices of the proposed model for different inputs: (a) only T1 MRI scans; (b) $T_1$ MRI scans with clinical information; (c) $T_1$ and $T_2$ MRI scans with clinical information; and (d) $T_1$, $T_2$, and $T_3$ MRI scans with clinical information, over ISPY-1 dataset.