Deep learning integrates histopathology and proteogenomics at a pan-cancer level

Abstract

We introduce a pioneering approach that integrates pathology imaging with transcriptomics and proteomics to identify predictive histology features associated with critical clinical outcomes in cancer. We utilize 2,755 H&E-stained histopathological slides from 657 patients across 6 cancer types from CPTAC. Our models effectively recapitulate distinctions readily made by human pathologists: tumor vs. normal (AUROC = 0.995) and tissue-of-origin (AUROC = 0.979). We further investigate predictive power on tasks not normally performed from H&E alone, including TP53 prediction and pathologic stage. Importantly, we describe predictive morphologies not previously utilized in a clinical setting. The incorporation of transcriptomics and proteomics identifies pathway-level signatures and cellular processes driving predictive histology features. Model generalizability and interpretability is confirmed using TCGA. We propose a classification system for these tasks, and suggest potential clinical applications for this integrated human and machine learning approach. A publicly available web-based platform implements these models.

Keywords: CPTAC; cancer imaging; cancer proteogenomics; computational pathology; molecular diagnostics.

Graphical abstract

Figure 1

Workflow, data split, and model performance (A) Overall workflow. Multi-resolution Panoptes models were trained on H&E slide images from six cancer types. Multi-CCA correlated proteomics, transcriptomics, and extracted imaging features from CNN models to reveal significant pathways and molecular signatures. (B) Per-slide level AUROCs of imaging-based prediction tasks with 95% confidence intervals.

Figure 2

Tissue-of-origin model performance and omics-integration (A) AUROC for each cancer type at per-slide level. (B) AUROC at per-tile level. (C) Features extracted from penultimate layer are separated with tSNE; each dot represents a tumor tile colored by tissue origin. (D) Feature extraction where each dot represents NAT tiles colored by tissue origin. (E) CCA canonical variate highlighting similarities between UCEC and LUAD samples. Line graphs represent standardized coefficients for subsets of imaging, gene, and proteome features. Each dot represents an image-proteogenomic paired sample. GO term enrichment assessed on subset of genes and proteome features with non-zero values in loading matrix. (F and G) Top and bottom images represent tiles with highest and lowest scores, respectively. Histopathology annotations reflect enriched GO terms.

Figure 3

Feature visualization and cross-testing of tumorigenesis models (A) Example UCEC slide with tumor tissue on left and normal tissue on right. (B) Prediction heatmap of example slide with hotter areas (red) highlighting tiles more likely to be tumor tissue. (C) CAM of example slide by tiles with hotter areas emphasizing the tumor tissue. (D–F) Feature extraction from tumorigenesis imaging model by tSNE; each dot represents a tile colored by prediction score, true label, and cancer type, respectively. (G) Example tiles of integrated saliency results highlighting accumulation of nuclei, with densest regions largely composed of stromal lymphoplasmacytic infiltrates. (H) Heatmap showing per-slide AUROCs of applying single cancer type trained models to the other cancer types.

Figure 4

Major canonical variates associated with tumorigenesis (A) Canonical variate with strongest correlation separating NAT/tumor samples across all six cancer types. (B) Tiles from highest-scoring regions show mitotic morphologies consistent with enriched transcriptomic and proteomic enrichment. (C) Tiles from lowest-scoring region. (D) Second canonical variate distinguishing NAT/tumor samples. (E and F) Tile scoring parallels enriched biological processes. Tile borders indicate scores; top-scoring regions (red) match tumorigenic areas with increased glycolytic activity, and bottom-scoring (blue) areas correspond with smooth muscle and blood vessel architectures.

Figure 5

Model performance and multi-omics assessment of grade and stage (A) Per-slide performance of models trained on tumor grade and disease stage. Numeric predictions represent expected value from softmax layer (${\sum }_{x=0}^{4}p\left(x\right)\left(x\right)$) where x represents grade or stage outcome). AUROC for each outcome denoted. (B) CCA canonical variate uniquely observed in grade analysis. Tiles with highest projected values (shown by more intense red borders) reflect regions with disorganized tumor nests lacking lumen formation and glandular regions with loss of basal nuclear polarity. Paler tile borders reflect lower projected values.

Figure 6

Performance, visualization, and feature extraction of biomarkers (A) One-tail Wilcoxon tests on prediction scores between positively and negatively labeled samples at per-tile level with significance levels. (B) Extraction and visualization of features learned by pan-cancer TP53 mutation model with tSNE. Reference plots of prediction scores and true labels on the right. (C) Canonical variate with strongest association between image and proteogenomic features. (D) Top tiles demonstrate highly cellular disordered regions correlating with TP53 mutated samples. (E) Bottom tiles (wild-type) highlight organized and well-differentiated regions. (F) Canonical variate correlating increased IL-1 activity with TP53 mutated samples. (G) Wild-type samples in canonical variate no. 3 highlight densely packed but relatively preserved tissue architectures. (H) Conversely, mutated samples reside in the bottom portion and show areas of increased immune infiltrate activity.

Figure 7

Panoptes Web (A) App workflow. (B) Boxplot assessment of probability scores and class outcomes, and individual tile probability visualization.