Multimodal analysis methods in predictive biomedicine

Abstract

For medicine to fulfill its promise of personalized treatments based on a better understanding of disease biology, computational and statistical tools must exist to analyze the increasing amount of patient data that becomes available. A particular challenge is that several types of data are being measured to cope with the complexity of the underlying systems, enhance predictive modeling and enrich molecular understanding. Here we review a number of recent approaches that specialize in the analysis of multimodal data in the context of predictive biomedicine. We focus on methods that combine different OMIC measurements with image or genome variation data. Our overview shows the diversity of methods that address analysis challenges and reveals new avenues for novel developments.

Keywords: Machine learning; Multi-omics; Multimodal modeling; Personalized medicine; Predictive modeling.

Graphical abstract

Fig. 1

Overview of multimodal data types and prediction tasks that are discussed in this review.

Fig. 2

Representation learning for multi-omics data integration. Low-dimensional latent variables (LVs, middle) are derived from multimodal, high-dimensional molecular data (omics layers, left). Based on different techniques including deep neural networks, autoencoders, or graph-based methods (integration method) and optionally leveraging existing prior knowledge, LVs are inferred such that they are associated with a clinical outcome of interest (right).

Fig. 3

Multimodal data fusion. Whole-slide images of a tissue (A) are segmented into smaller patches (B). (C) Image patch and genomic feature-specific embeddings are learned. (D) Multimodal-guided embeddings and concatenation allow prediction of survival or disease risk.