A whole-slide foundation model for digital pathology from real-world data

Abstract

Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles^1-3. Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context⁴. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet⁵ method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data⁶. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision-language pretraining for pathology^7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.

Fig. 1. Overview of Prov-GigaPath.

a, Flow chart showing the model architecture of Prov-GigaPath. Prov-GigaPath first serializes each input WSI into a sequence of 256 × 256 image tiles in row-major order and uses an image tile-level encoder to convert each image tile into a visual embedding. Then Prov-GigaPath applies a slide-level encoder based on the LongNet architecture to generate contextualized embeddings, which can serve as the basis for various downstream applications. b, Image tile-level pretraining using DINOv2. c, Slide-level pretraining with LongNet using masked autoencoder. [CLS] is the classification token.

Fig. 2. Gene mutation prediction.

a−j, Bar plots comparing the AUROC and AUPRC scores of Prov-GigaPath and competing methods on pan-cancer 18-biomarker (a,f), LUAD-specific 5-gene mutation prediction (b,g), pan-cancer 5-gene mutation prediction (c,h), LUAD-specific 5-gene mutation prediction on TCGA (d,i) and pan-cancer TMB prediction (e,j). k, Bar plot showing AUROC for each gene on LUAD-specific five-gene mutation prediction on TCGA. a−k, Data are mean ± s.e.m. across n = 10 independent experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. l, Comparison of AUROC scores for individual biomarkers in pan-cancer 18-biomarker predictions.

Fig. 3. Comparison on cancer subtyping.

a–f, Bar plots comparing cancer subtyping performance in terms of AUROC (a,c,e) and balanced accuracy (b,d,f) on nine cancer types. Data are mean ± s.e.m. across n = 10 independent experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. BACC, balanced accuracy. BRCA, breast invasive carcinoma; CNS, central nervous system; COADREAD, colorectal adenocarcinoma; DIFG, diffuse intrinsic pontine glioma; EGC, early gastric cancer; HB, hepatobiliary; NSCLC, non-small cell lung cancer; OVT, ovarian cancer; RCC, renal cell cancer.

Fig. 4. Comparison on image–report alignment.

a, Flow chart showing the fine-tuning of Prov-GigaPath using pathology reports. Real-world pathology reports are processed using GPT-3.5 from OpenAI to remove information irrelevant to cancer diagnosis. We performed the CLIP-based contrastive learning to align Prov-GigaPath and PubMedBERT. b, The fine-tuned Prov-GigaPath can then be used to perform zero-shot cancer subtyping and mutation prediction. The input of Prov-GigaPath is a sequence of tiles segmented from a WSI, and the inputs of the text encoder PubMedBERT are manually designed prompts representing cancer types and mutations. Based on the output of Prov-GigaPath and PubMedBERT, we can calculate the probability of the input WSI being classified into specific cancer subtypes and mutations. c, Bar plots comparing zero-shot subtyping performance on NSCLC and COADREAD in terms of BACC, precision and f₁. d, Bar plots comparing the performance on mutation prediction using the fine-tuned model for six genes. c,d, Data are mean ± s.e.m. across n = 50 experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. e, Scatter plots comparing the performance between Prov-GigaPath and MI-Zero in terms of BACC on zero-shot cancer subtyping. Each dot indicates one trial with a particular set of text query formulations.

Extended Data Fig. 1. Comparison on Pan-cancer 18-biomarker prediction.

Bar plot showing the AUPRC score for each biomarker on the 18-biomarker prediction by Prov-GigaPath and competing methods.

Extended Data Fig. 2. Comparison on LUAD 5-gene mutation prediction.

Bar plots showing AUROC and AUPRC scores for predicting each gene mutation on LUAD 5-gene mutation prediction. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.

Extended Data Fig. 3. Comparison on Pan-cancer 5-gene mutation prediction.

Bar plots showing AUROC and AUPRC scores for predicting each gene mutation on Pan-cancer 5-gene mutation prediction. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.

Extended Data Fig. 4. Comparison on LUAD 5-gene mutation prediction in TCGA.

Bar plots showing AUPRC scores for predicting each gene mutation on LUAD 5-gene mutation prediction in TCGA. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.

Extended Data Fig. 5. Comparison on mutation prediction on new colorectal patients.

Bar plots showing AUROC and AUPRC scores for predicting 5-gene mutation and TMB status on new patients from Providence. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.

Extended Data Fig. 6. Comparison between pretraining the same model using Prov-Path and TCGA.

a-b, Bar plots showing the AUROC (a) and AURPC (b) on LUAD 5-gene mutation prediction in TCGA using models trained on Prov-Path and TCGA. Prov-GigaPath is GigaPath trained on Prov-Path. GigaPath-TCGA is GigaPath trained on TCGA. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms GigaPath-TCGA, with one-sided Wilcoxon test.

Extended Data Fig. 7. Comparison between GigaPath trained using Prov-Path and HIPT trained using Prov-Path on mutation prediction.

a-j: Bar plots showing the AUROC (a-e) and AURPC (f-j) of mutation prediction tasks by Prov-GigaPath and HIPT-Prov-Path. HIPT-Prov-Path indicates HIPT pretrained on Prov-Path. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the HIPT-Prov-Path, with one-sided Wilcoxon test.

Extended Data Fig. 8. Comparison between GigaPath trained using Prov-Path and HIPT trained using Prov-Path on cancer subtyping.

a-f, Bar plots showing the AUROC (a,c,e) and BACC (b,d,f) of cancer subtyping tasks by Prov-GigaPath and HIPT-Prov-Path. HIPT-Prov-Path indicates HIPT pretrained on Prov-Path. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the HIPT-Prov-Path, with one-sided Wilcoxon test.

Extended Data Fig. 9. Alignment between pathology reports and images.

a-d, Bar plots showing the performance of f₁ (a), Precision (b), AUROC (c) and AUPRC (d) using fine-tuned Prov-GigaPath to predict mutations in the zero-shot learning setting. The error bars show the standard error across n = 50 experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test. e, Scatter plots comparing Prov-GigaPath and MI-Zero on cancer subtyping prediction and mutation prediction in terms of balanced accuracy (BACC).