Multimodal deep learning to predict prognosis in adult and pediatric brain tumors

Abstract

Background: The introduction of deep learning in both imaging and genomics has significantly advanced the analysis of biomedical data. For complex diseases such as cancer, different data modalities may reveal different disease characteristics, and the integration of imaging with genomic data has the potential to unravel additional information than when using these data sources in isolation. Here, we propose a DL framework that combines these two modalities with the aim to predict brain tumor prognosis.

Methods: Using two separate glioma cohorts of 783 adults and 305 pediatric patients we developed a DL framework that can fuse histopathology images with gene expression profiles. Three strategies for data fusion were implemented and compared: early, late, and joint fusion. Additional validation of the adult glioma models was done on an independent cohort of 97 adult patients.

Results: Here we show that the developed multimodal data models achieve better prediction results compared to the single data models, but also lead to the identification of more relevant biological pathways. When testing our adult models on a third brain tumor dataset, we show our multimodal framework is able to generalize and performs better on new data from different cohorts. Leveraging the concept of transfer learning, we demonstrate how our pediatric multimodal models can be used to predict prognosis for two more rare (less available samples) pediatric brain tumors.

Conclusions: Our study illustrates that a multimodal data fusion approach can be successfully implemented and customized to model clinical outcome of adult and pediatric brain tumors.

Fig. 1. Overview of used datasets and samples.

a Adult brain tumor The Cancer Genome Atlas (TCGA) cohort (N = 783) consisting of both low-grade glioma (LGG) (N = 426) and glioblastoma (GBM) (357) samples. For the LGG samples, both histopathology images and RNA-sequencing data were available. While histopathology images were available for all GBM samples, 158 of these had RNA-sequencing, and 199 microarray expression data. b Adult Clinical Proteomic Tumor Analysis Consortium (CPTAC)-GBM cohort with histopathology and RNA-sequencing data (N = 97). c Pediatric Brain Tumor Atlas (PBTA) Kids First cohort (N = 305). This cohorts consists of four tumor subtypes: LGG and high-grade (HG) astrocytoma (combined in one Glioma group, N = 198), ependymoma (N = 47), and Medulloblastoma (N = 60). Both histopathology images and RNA-sequencing data were available for all samples of this pediatric cohort.

Fig. 2. Feature extraction and survival prediction for histopathological and expression data.

a ResNet-50 feature extraction flow for pathology image patches. b Multi-Layer Perceptron (MLP) feature extraction flow for gene expression data. Both extraction flows produce a modality-specific feature vector of dimension 2048 × 1.

Fig. 3. Visualization of the three data fusion strategies to integrate histopathological and expression data.

a Early or feature fusion. b Late fusion, and c Joint fusion.

Fig. 4. Kaplan–Meier curves of the adult glioma (N = 156) and pediatric glioma (N = 39) test set.

a Histopathology model. b Gene expression model. c Early or Feature fusion model. d Joint fusion model. e Late fusion model.

Fig. 5. Boxplots of model performance for each model strategy on the adult glioma cohort.

a Composite Score (CS) distribution on cross-validation (CV) validation sets (N = 63). b CS distribution of each CV fold on the test set (N = 156). (*P value <0.05, **P value <0.01 and ***P value <0.005; pairwise Wilcoxon signed-rank test).

Fig. 6. Visualization of pathway importance with respect to survival predictions for the pediatric glioma cohort.

Pathways are ranked from top to bottom based on the sum of the absolute gradients across all samples. Negative gradients contribute to a lower risk score, while positive gradients lead to a higher risk score. a Top 15 pathways of the unimodal gene expression model (RNA only). b Top 15 pathways of the multimodal joint fusion model (histopathology + RNA data).

Fig. 7. Interpretability analysis of histopathology model with respect to survival predictions and cell type distributions.

a Examples of high-risk and low-risk patches for a bad survival sample. b Examples of high-risk and low-risk patches for a good survival sample. Left panel = original 224 × 224 patch, scale bars (white insets), 20 µm; middle panels = overlayed saliency map for joint fusion and histopathology (FFPE only) models, Right panel = cell segmentation predicted by HoverNet. c Quantitative analysis of cell type composition in extracted 224 × 224 patches of bad survival and good survival samples of the adult glioma test set. (***P value < 2.2e-16, two sample t test, high-risk N = 78, low-risk N = 78).