Biology-guided deep learning predicts prognosis and cancer immunotherapy response

Abstract

Substantial progress has been made in using deep learning for cancer detection and diagnosis in medical images. Yet, there is limited success on prediction of treatment response and outcomes, which has important implications for personalized treatment strategies. A significant hurdle for clinical translation of current data-driven deep learning models is lack of interpretability, often attributable to a disconnect from the underlying pathobiology. Here, we present a biology-guided deep learning approach that enables simultaneous prediction of the tumor immune and stromal microenvironment status as well as treatment outcomes from medical images. We validate the model for predicting prognosis of gastric cancer and the benefit from adjuvant chemotherapy in a multi-center international study. Further, the model predicts response to immune checkpoint inhibitors and complements clinically approved biomarkers. Importantly, our model identifies a subset of mismatch repair-deficient tumors that are non-responsive to immunotherapy and may inform the selection of patients for combination treatments.

Fig. 1. Study design for the development and validation of a deep learning model to predict TME classes and disease-free survival.

Patients in the training (SMU-1) cohort and internal validation (SMU-2, 3) cohorts were recruited from Southern Medical University, Guangzhou, China. Patients in the external validation cohorts were recruited from Sun Yat-sen University Cancer Center (SYSUCC-1, 2), Guangzhou, China and Stanford University Medical Center, Palo Alto, USA. Patients in the immunotherapy cohort were recruited from Southern Medical University, Guangzhou, China and Guangdong Provincial Hospital of Chinese Medicine, Guangzhou, China. Both CT images and IHC-stained slides were available for patients in the SMU-1 training cohort, SMU-2 and SYSUCC-1 validation cohorts, which were used for evaluating the model’s accuracy for TME prediction. All patients had preoperative CT scans and outcomes available, which were used for testing the model’s prognostic and predictive value. CT: computer tomography; IHC: immunohistochemistry. SMU: Southern Medical University; SYSUCC: Sun Yat-sen University Cancer Center. TME: tumor microenvironment. Chemo: Chemotherapy.

Fig. 2. Proposed deep learning model and visualization, prediction for representative cases.

A Architecture of the proposed multi-task deep convolutional neural network to simultaneously classify TME and predict prognosis from CT image; (B) CT images and corresponding feature maps along with the predicted TME classes and survival scores for four representative cases, where each row corresponds to a patient with TME classes 1–4 defined by IHC. TME classes were correctly predicted for all four cases; predicted survival scores were also consistent with the actual patient outcome. TME tumor microenvironment.

Fig. 3. Accuracy of the deep learning model to assess TME classes.

A Receiver operator characteristic (ROC) curves and (B) confusion matrices in the training SMU-1 cohort, internal validation SMU-2 cohort, and external validation SYSUCC-1 cohort. The ROC curves show the one-vs-others comparison. The confusion matrices show the pair-wise comparison; diagonal: number of cases correctly classified; off-diagonal: number of cases incorrectly classified. TME tumor microenvironment, SMU Southern Medical University, SYSUCC Sun Yat-sen University Cancer Center, AUC area under the curves. Source data are provided as a Source Data file.

Fig. 4. Kaplan-Meier analyses of disease-free survival (DFS) and overall survival (OS) according to the model-predicted survival score in patients with gastric cancer.

A Training cohort SMU-1 (n = 348), (B) SMU-2 validation cohort (n = 202), (C) SMU-3 validation cohort (n = 636), (D) SYSUCC-1 validation cohort (n = 125), (E) SYSUCC-2 validation cohort (n = 1063), (F) Stanford validation cohort (n = 123). Comparisons of the survival curves were performed with a two-sided log-rank test. SMU Southern Medical University, SYSUCC Sun Yat-sen University Cancer Center, HR Hazard ratio. Source data are provided as a Source Data file.

Fig. 5. Prognosis prediction using the deep learning model and clinicopathologic risk factors.

A Accuracy of prediction for disease-free survival (DFS) using clinicopathologic variable and the deep learning model (n = 2025). The center lines within the boxes represent the mean AUC value, the bounds of boxes represent the interquartile range (IQR) and the whiskers represent the 95% confidence intervals. B Relative variable contribution to prediction of DFS using the χ² proportion test for clinicopathologic variables only in all patients (n = 2025); for clinicopathologic variables and DLS in all patients as well as patients with stage II and III disease. C Kaplan-Meier analysis of DFS according to the deep learning model-predicted survival score within each stage in the validation cohorts (n = 2148). D Kaplan-Meier analysis of DFS according to the nomogram combining deep learning model and clinicopathologic risk factors in the validation cohorts (n = 2025). For statistical comparisons among different groups, a two-tailed t test (unpaired) was used. Comparisons of the survival curves were performed with a two-sided log-rank test. DLS deep learning survival score. **P < 0.001, ***P < 0.0001, ****P < 0.00001. Source data are provided as a Source Data file.

Fig. 6. Relationship between the TME class groups and benefit from adjuvant chemotherapy in matched patients with stage II and III gastric cancer.

Kaplan-Meier curves of disease-free survival (DFS) for patients stratified by the receipt of chemotherapy. A TME class 1 (n = 274), TME class 2 (n = 382), TME class 3 (n = 456), TME class 4 (n = 520). B Forest plot for the effect of chemotherapy vs. no chemotherapy on DFS among stage II and III patients. Comparisons of the above survival curves were performed with a two-sided log-rank test. P values reported in (B) are two-tailed from Cox proportional hazard regression analyses. Blue dot represents the HR value. Error bars represent the 95% confidence intervals. TME tumor microenvironment, DLS deep learning survival score, TME tumor microenvironment, Chemo Chemotherapy, HR, Hazard ratio. Source data are provided as a Source Data file.

Fig. 7. Relationship between the deep learning model and benefit from adjuvant chemotherapy in matched patients with stage II and III gastric cancer.

Kaplan-Meier curves of disease-free survival (DFS) for patients stratified by the receipt of chemotherapy. A TME class 2 & DLS Low (n = 172), TME class 2 & DLS High (n = 178). B TME class 3 & DLS Low (n = 230), TME class 3 & DLS High (n = 226). C Forest plot for the effect of chemotherapy vs. no chemotherapy on DFS among TME Class 2/3 patients with stage II and III disease. Comparisons of the above survival curves were performed with a two-sided log-rank test. P values reported in (C) are two-tailed from Cox proportional hazard regression analyses. Blue dot represents the HR value. Error bars represent the 95% confidence intervals. DLS deep learning survival score, TME tumor microenvironment, Chemo Chemotherapy, HR Hazard ratio. Source data are provided as a Source Data file.

Fig. 8. Performance of the deep learning model in predicting response and outcomes in patients treated with anti-PD-1 immunotherapy.

A Response rates in patients of four TME classes predicted by the deep learning model; (B), Progression-free survival in patients of four predicted TME classes; (C), Receiver operator characteristic (ROC) curves of the predicted TME classes, CPS and composite models combining TME classes and CPS for predicting immunotherapy response (n = 296); (D), AUC values of the predicted TME classes, CPS and composite models combining TME classes and CPS for predicting immunotherapy response (n = 296); (E), Forest plot for the multivariate logistic regression analysis for objective response; (F), Decision tree combining the predicted TME classes and CPS. Comparisons of the survival curves were performed with a two-sided log-rank test. Comparisons of the bar plot were performed with a two-sided t (unpaired) test. P values reported in (E) are two-tailed from logistic regression analyses. Blue dot represents the HR value. Error bars in (D) and (E) represent the 95% confidence intervals. TME tumor microenvironment, AUC area under the receiver operator characteristic curve, CPS combined positive score of PDL1 expression, OR objective response (complete and partial response), SD stable disease, PD progressive disease. Source data are provided as a Source Data file.