Multimodal data fusion for cancer biomarker discovery with deep learning

Abstract

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Fig. 1:. Generation and processing of routinely collected biomedical modalities in oncology.

Prior to data fusion, different steps are needed to go from the raw data to workable data representations for each modality, e.g. EHRs, molecular data and medical images.

Fig. 2:. Overview of different fusion strategies for multimodal data.

a) Raw data is processed into workable formats. b) For each modality features are extracted using dedicated encoder algorithms. c) Early fusion. d) Intermediate fusion. e) Late fusion.

Fig. 3:. Examples of model interpretability methods for histopathology and gene expression.

Histopathology: a) Examples of informative tiles for predicting the presence of TP53 mutations from histopathology images in prostate cancer (unpublished data). b) Visualisation of regions within tiles most relevant to the prediction, derived via Grad-CAM. c) Individual cells within informative tiles are segmented and classified by Hover-Net. For a fine-grained interpretation of relevant cells (black annotations), pertinent cells within the tile are encircled by calculating the contours from regions highlighted by Grad-CAM. Gene Expression: d) Examples of SHAP visualisation of hypothetical gene importance according to unimodal model (top) and joint multimodal model (bottom) for cancer survival prediction. e) Example of pathway importance visualisation based on the respective gene SHAP-values in unimodal (top) versus joint multimodal (bottom) models with respect to cancer survival prediction.