Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging

Abstract

Purpose: Tumors are continuously evolving biological systems, and medical imaging is uniquely positioned to monitor changes throughout treatment. Although qualitatively tracking lesions over space and time may be trivial, the development of clinically relevant, automated radiomics methods that incorporate serial imaging data is far more challenging. In this study, we evaluated deep learning networks for predicting clinical outcomes through analyzing time series CT images of patients with locally advanced non-small cell lung cancer (NSCLC).Experimental Design: Dataset A consists of 179 patients with stage III NSCLC treated with definitive chemoradiation, with pretreatment and posttreatment CT images at 1, 3, and 6 months follow-up (581 scans). Models were developed using transfer learning of convolutional neural networks (CNN) with recurrent neural networks (RNN), using single seed-point tumor localization. Pathologic response validation was performed on dataset B, comprising 89 patients with NSCLC treated with chemoradiation and surgery (178 scans).

Results: Deep learning models using time series scans were significantly predictive of survival and cancer-specific outcomes (progression, distant metastases, and local-regional recurrence). Model performance was enhanced with each additional follow-up scan into the CNN model (e.g., 2-year overall survival: AUC = 0.74, P < 0.05). The models stratified patients into low and high mortality risk groups, which were significantly associated with overall survival [HR = 6.16; 95% confidence interval (CI), 2.17-17.44; P < 0.001]. The model also significantly predicted pathologic response in dataset B (P = 0.016).

Conclusions: We demonstrate that deep learning can integrate imaging scans at multiple timepoints to improve clinical outcome predictions. AI-based noninvasive radiomics biomarkers can have a significant impact in the clinic given their low cost and minimal requirements for human input.

Figure 1:. Serial Patient Scans.

Representative computed tomography (CT) images of stage III non-surgical NSCLC patients before radiation therapy and one, three, and six months post radiation therapy. A single click seed point identifies the input image patch (defined by the dotted white line) were inputted into the neural network.

Figure 2:. Analysis Design.

Depiction of the deep learning based workflow with two datasets and additional comparative models. Dataset A included patients treated with chemotherapy and definitive radiation therapy, and was used to train and fine tune a ResNet convolutional neural network (CNN) combined with a recurrent neural network (RNN) for predictions of survival. A separate test set from this cohort was used to assess performance and compared with the performance of radiographic and clinical features. Dataset B included patients treated with chemotherapy and surgery. This cohort was used as an additional test set to predict pathological response, and the model predictions were compared to the change in volume.

Figure 3:. Deep Learning Architectures.

The neural architecture includes ResNet convolutional neural networks (CNN) merged with a recurrent neural network (RNN), and was trained on baseline and follow-up scans. The input axial slices of 50 × 50 mm² centered on, 5 mm proximal to and 5 mm distal to the selected seed point. Deep learning networks are trained on natural RGB images, and thus need 3 image slices for input. The outputs of each CNN model are input into the RNN, with a gated recurrent unit (GRU) for time-varying inputs. Masking was performed on certain inputs of the CNN so that the recurrent network takes missed scans into account. The final softmax layer provides the prediction.

Figure 4:. Performance Deep Learning Biomarkers on Validation Datasets.

The deep learning models were evaluated on an independent test set for performance. The two-year overall survival Kaplan Meier curves were performed with median stratification (derived from the training set) of the low and high mortality risk groups with no follow-up, or up to three follow-ups at one, three and six months post treatment for Dataset A (72 definitive patients in the independent test set, log-rank test p < 0.05 for > one follow-up).

Figure 5:. Pathological Response Prediction Validation.

Model probability and the change in volume after radiation therapy was used for the prediction of pathologic response. The CNN survival model significantly stratified response and gross residual disease in the second test set Dataset B, comparable predictions were found with change in tumor volume, and the combination of the two parameters (n=89, Wilcoxon p < 0.05).