Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Abstract

Cancer diagnosis, prognosis, mymargin and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment. Code and trained models are made available at: https://github.com/mahmoodlab/PathomicFusion.

Fig. 1:. Pathomic Fusion:

An integrated framework for multimodal fusion of histology and genomic features for survival outcome prediction and classification. Histology features may be extracted using CNNs, parameter efficient GCNs or a combination of the two. Unimodal networks for the respective image and genomic features are first trained individually for the corresponding supervised learning task, and then used as feature extractors for multimodal fusion. Multimodal fusion is performed by applying an gating-based attention mechanism to first control the expressiveness of each modality, followed by the Kronecker product to model pairwise feature interactions across modalities.

Fig. 2:

Graph Convolutional Network for learning morphometric cell features from histology images. We represent cells in histology tissue as nodes in a graph, where cells are isolated using a deep learning-based nuclei segmentation algorithm and the connections between cells are made using KNN. Features for each cell are initialized using handcrafted features as well as deep features learned using contrastive predictive coding. The aggregate and combine functions are adopted from the GraphSAGE architecture, with the node masking and hierarchical pooling strategy adopted from SAGEPool.

Fig. 3:. Pathomic Fusion Applied to Glioblastoma and Lower Grade Glioma.

A. Glioma hazard distributions amongst shorter vs. longer surviving uncensored patients and molecular subtypes for Histology CNN and Pathomic Fusion. Patients are defined as shorter surviving if patient death is observed before 5 years of the first follow-up (shaded red), and longer surviving if patient death is observed after 5 years of the first follow-up (shaded blue). Pathomic Fusion predicts hazard in more concentrated clusters than Histology CNN, while the distribution of hazard predictions from Histology CNN have longer tails and are more varied across molecular subtypes. In analyzing the types of glioma in the three high density regions revealed from Pathomic Fusion, we see that these regions corroborate with the WHO paradigm for stratifying patients into IDHwt ATC, IDHmut ATC, and ODG (Appendix C, Table IV). B. Kaplan-Meier comparative analysis of using grade, molecular subtype, Histology CNN and Pathomic Fusion in stratifying patient outcomes. Hazard predictions from Pathomic Fusion show better stratification of mid-to-high risk patients than Histology CNN, and low-to-mid risk patients than molecular subtyping, which follows the WHO paradigm. Low / intermediate / high risk are defined by the 33-66-100 percentile of hazard predictions. Overlayed Kaplan-Meier estimates of our network predictions with WHO Grading is shown in the supplement (Appendix C, Fig. 9).

Fig. 4:. Pathomic Fusion Applied to Clear Cell Renal Cell Carcinoma.

CCRCC hazard distributions amongst shorter vs. longer surviving uncensored patients for Histology CNN and Pathomic Fusion. Patients are defined as shorter surviving if patient death is observed before 3.5 years of the first follow-up (shaded red), and longer surviving if patient death is observed after 3.5 years of the first follow-up (shaded blue). Pathomic Fusion was observed to able to stratify longer and shorter surviving patients better than Histology CNN, exhibiting a bimodal distribution in hazard prediction. Overlayed Kaplan-Meier estimates of our network predictions with WHO Grading is shown in the supplement (Appendix C, Fig. 10).

Fig. 5:

Multimodal interpretability by Pathomic Fusion in glioma. A. Local explanation of histology image, cell graph, and genomic modalities for individual patients of three molecular subtypes. In IDHwt ATC, the network detects endothelial cells of the microvascular proliferation in the histology image, while the cell graph localizes glial cells between the microvasculature. In IDHmut ATC, we observe similar localization of tumor cellularity in both the histology image and cell graph, however, attribution direction for IDH is flipped to have positive impact on survival. In ODG, we observe both modalities localizing towards different regions containing “fried egg cells” that are canonical in ODG. For each of these patients, local explanation reveals the most important genomic features used for prediction. B. Global explanation of top 20 genomic features for each molecular subtype in glioma. Canonical oncogenes in glioma such as IDH, PTEN, MYC and CDKN2A are attributed highly as being important for risk prediction.

Fig. 6:

Multimodal interpretability by Pathomic Fusion in CCRCC. A. Local explanation of histology image, cell graph, and genomic modalities for two longer and shorter surviving patients. In the longer surviving patient, Pathomic Fusion localizes cells without obvious nucleoli in both the histology image and cell graph, which suggests lower-grade CCRCC and lower risk. In the shorter surviving patient, we observe Pathomic Fusion attending to large cells with prominent nucleoli and eosinophilic-to-clear cytoplasm in the cell graph, and the “chicken-wire” vasculature pattern in the histology image that is characteristic of higher-grade CCRCC. Cells without clear cytoplasms are noticeably missed in both modalities for shorter survival. For each of these patients, local explanation reveals the most important genomic features used for prediction. B. Global explanation of top 20 genomic features for longer surviving, shorter surviving, and all patients in CCRCC. Genes such as CYP3A7, DDX43 and PITX2 are attributed highly as being important for risk prediction, which have linked to cancer predisposition and tumor progression in CCRCC and other cancers.