Pan-cancer image-based detection of clinically actionable genetic alterations

Abstract

Molecular alterations in cancer can cause phenotypic changes in tumor cells and their micro-environment. Routine histopathology tissue slides - which are ubiquitously available - can reflect such morphological changes. Here, we show that deep learning can consistently infer a wide range of genetic mutations, molecular tumor subtypes, gene expression signatures and standard pathology biomarkers directly from routine histology. We developed, optimized, validated and publicly released a one-stop-shop workflow and applied it to tissue slides of more than 5000 patients across multiple solid tumors. Our findings show that a single deep learning algorithm can be trained to predict a wide range of molecular alterations from routine, paraffin-embedded histology slides stained with hematoxylin and eosin. These predictions generalize to other populations and are spatially resolved. Our method can be implemented on mobile hardware, potentially enabling point-of-care diagnostics for personalized cancer treatment. More generally, this approach could elucidate and quantify genotype-phenotype links in cancer.

Extended Data Fig. 1

Extended Data Fig. 2

Extended Data Fig. 3

Extended Data Fig. 4

Extended Data Fig. 5

Extended Data Fig. 6

Extended Data Fig. 7

Extended Data Fig. 8

Extended Data Fig. 9

Extended Data Fig. 10

Fig. 1. Deep learning workflow for prediction of molecular features from histology images.

We describe a comprehensive method pipeline for prediction of molecular features directly from histological images. (a) Training of the deep learning system comprised six steps. Step 1: Patient cohorts were randomly split into three partitions for cross-validation of deep classifiers. Step 2: The tumor region on each whole slide image (WSI) was tessellated into tiles. Step 3: Up to 500 randomly chosen tiles were collected. Step 4: Tiles from patients in the training partitions were collected, classes were equalized by random undersampling. Step 5: All training tiles were used to train a deep neural network (pre-trained on a non-medical task). Step 6: Classification performance was evaluated on patients from the test partition. (b) For patient-level inference of molecular labels in patients not seen during training, three successive steps were used. Step 1: Tiles were generated from the tumor region on WSI. Step 2: A prediction was made for each tile. Step 3: Tile-level class predictions were pooled on a patient level. (c) Hyperparameters of the deep learning system were optimized in a benchmark task (prediction of microsatellite instability status [MSI] in colorectal cancer). The opacity of each point corresponds to the number of trainable layers. Shufflenet, a lightweight neural network architecture was selected as a highly efficient network model. (d) This workflow was subsequently applied for prediction of four types of molecular features across 14 cancer types. In particular, this included genetic mutations. The distribution of the 20 most common among all analyzed mutations is shown for each tumor type.

Fig. 2. Inference of genetic mutations from histological images.

A deep learning system was trained to predict mutational status (mutated or wild-type) of relevant genes in 14 cancer type and was evaluated by cross-validation. All mutations, including variants of unknown significance, were included in the ‘mutated’ class. For each gene, patient-level test set performance is shown as area under the receiver operating curve (AUROC) with two-sided t-test p-value for prediction scores corrected for multiple testing (false detection rate, FDR). The significance level of 0.05 is marked with a line and the distribution of p-values in each panel is shown as a density plot. P values smaller than 10⁻⁵ are set to 10⁻⁵. On the right-hand side of each panel, a kernel density estimate shows the distribution of all plotted data points. “n” denotes the number of patients with available genetic information and matched histology images in each tumor type. (a-d) In lung adenocarcinoma, colorectal cancer, breast cancer and gastric cancer, a number of relevant genes were significantly predictable from histology alone, including key oncogenic drivers such as TP53, BRAF and MTOR. (e-n) In all other tested tumor types, mutational status was predictable for some genes, with notable examples including KRAS in pancreatic cancer, CTNNB1 in hepatocellular carcinoma and TP53 and CASP8 in head and neck cancer.

Fig. 3. Inference of putative oncogenic drivers from histological images.

A deep learning system was trained to predict oncogenic driver genes from histology. Only putative and confirmed drivers were included and variants of unknown significance were pooled with the “wild type” class. On the right-hand side of each panel, a kernel density estimate shows the distribution of all plotted data points. “n” denotes the number of patients with available genetic information and matched histology images in each tumor type. The layout of this figure corresponds to Fig. 2. (a-n) This process uncovered significant predictability of multiple oncogenic drivers, including EGFR, BRAF and TP53.

Fig. 4. Inference of molecular subtypes, gene expression signatures and standard biomarkers directly from histology.

In addition to prediction of singlegene mutations, the capability of deep learning to infer high-level molecular features was systematically assessed. (a-d) In lung, colorectal, breast and gastric cancer, gene expression signatures (such as TCGA molecular subtype in any tumor type) and standard of care features (such as hormone receptor status in breast cancer) were highly predictable from histology alone, as shown by the distribution of two-sided t-test false-detection rate (FDR)-corrected p-values as visualized by a kernel density estimate. Individual data points are shown in Extended Data Figures 2-6. (e-h) Gene expression signatures for Proliferation (Prolif), Wound Healing (WoundHeal), Macrophage infiltration (Mcrphg), Homologous Repair Deficiency (HRD), CD8-positive Lymphocyte (LymCD8), TCGA molecular subtypes (LUAD 1-6), pan-gastrointestinal (GI) molecular subtypes, consensus molecular subtypes (CMS), PAM50 subtypes and other key molecular features were highly predictable across multiple tumor types. Error bars show patient-level AUROC with bootstrapped confidence intervals, the marker denotes the mean, * denotes two-sided t-test FDR-corrected p-value< 0.05. „n“ refers to the number of patients. (i-j) Standard of care biomarkers including estrogen and progesterone receptor (ER and PR) status in breast cancer, pathologic subtype and microsatellite instability (MSI) were highly predictable from routine histology alone by deep learning.

Fig. 5. Explainability of deep learning-based analysis of histological images.

Deep learning-based predictions were visualized through genotype maps and comparison of highly ranked image tiles. (a-e) Prediction maps for consensus molecular subtype (CMS) in colorectal cancer show spatially resolved prediction scores, unveiling intratumor heterogeneity of predicted genotype. As a generic tool, this visualization approach allows to identify spatial regions associated with a molecular feature. In this patient, the correct prediction of CMS4 correctly show that deep learning robustly predicts CMS from histology alone while highlighting potential intratumor heterogeneity (f-i) For each of the CMS classes, the most highly scored test set tiles are shown, enabling correlation of deep learning-predictions with histopathological features at high resolution. In this case, highly predicted CMS1 tiles contain numerous tumorinfiltrating lymphocytes while predicted CMS4 tiles contain abundant stroma, consistent with previous studies. (j-k) Highly scored tiles in the external test cohort DACHS for prediction of BRAF mutant and wild type (l-m) and CpG-island methylator phenotype (CIMP) high or non-CIMP.

Fig. 6. Highest scoring image tiles for molecular features in gastric cancer.

(a-b) Highest scoring tiles in highest scoring patients corresponding to AMER1 mutational status in the TCGA-STAD data set. (c-d) Tiles corresponding to MTOR mutational status. (e-f) Tiles corresponding to high or low values of a proliferation signature. (a-b) Tiles corresponding to hypermutated samples.