A comprehensive evaluation of histopathology foundation models for ovarian cancer subtype classification

Abstract

Histopathology foundation models show great promise across many tasks, but analyses have been limited by arbitrary hyperparameters. We report the most rigorous single-task validation study to date, specifically in the context of ovarian carcinoma morphological subtyping. Attention-based multiple instance learning classifiers were compared using three ImageNet-pretrained encoders and fourteen foundation models, each trained with 1864 whole slide images and validated through hold-out testing and two external validations (the Transcanadian Study and OCEAN Challenge). The best-performing classifier used the H-optimus-0 foundation model, with balanced accuracies of 89%, 97%, and 74%, though UNI achieved similar results at a quarter of the computational cost. Hyperparameter tuning the classifiers improved performance by a median 1.9% balanced accuracy, with many improvements being statistically significant. Foundation models improve classification performance and may allow for clinical utility, with models providing a second opinion in challenging cases and potentially improving the accuracy and efficiency of diagnoses.

Fig. 1. Classification model pipeline.

Attention-based multiple instance learning (ABMIL) classifier for ovarian cancer subtyping, showing the classification of a high-grade serous carcinoma (HGSC).

Fig. 2. Ovarian cancer subtyping results.

The mean and 95% confidence interval generated by 10,000 iterations of bootstrapping for each metric. Blue indicates ImageNet-pretrained feature extractors and orange indicates histopathology foundation models. Hold-out testing and external validation results are based on an ensemble of cross-validation models. Precise values are provided in Supplementary Tables 3–6.

Fig. 3. Optimal confusion matrices.

The confusion matrix from each validation for the optimal ABMIL classifier with features from the H-optimus-0 foundation model. Correct classifications are indicated in green.

Fig. 4. Model inference times.

The average inference time per WSI for each model, including tissue patch extraction, feature encoding, and ABMIL classification time.

Fig. 5. Preprocessing analysis results.

Comparison of the balanced accuracy for each ImageNet-pretrained feature extractor (blue), the seven ResNet50 models with varied preprocessing techniques (green), and the three worst-performing (RN18-Histo, RN50-Histo, and CTransPath) and the single best-performing foundation models (H-optimus-0) in (a) cross-validation, (b) hold-out testing, (c) external validation on the Transcanadian Study dataset, (d) external validation on the OCEAN Challenge dataset. For validations (b)–(d), predictions were ensembled from the five cross-validation models. Results reported as the mean and 95% confidence interval generated by 10,000 iterations of bootstrapping. Precise values and other metrics are presented in Supplementary Tables 12–15.

Fig. 6. Validation losses.

The average validation loss from five-fold cross-validation for each model across each hyperparameter tuning iteration.

Fig. 7. Results of hyperparameter tuning.

The balanced accuracy compared for each ABMIL model trained with the default hyperparameters (pink) and the tuned hyperparameters (blue) in (a) cross-validation, (b) hold-out testing, (c) external validation on the Transcanadian Study dataset, and (d) external validation on the OCEAN Challenge dataset. For validations (b)–(d), predictions were ensembled from the five cross-validation models. *Indicates a significant difference in the paired t-test at the 5% significance level.

Fig. 8. Accuracy compared to efficiency.

Balanced accuracy results for each histopathology foundation model-based classifier in each validation shown in relation to the number of model parameters and number of WSIs used in the pretraining of the foundation model. The line of best fit and the corresponding coefficient of determination (R²) are provided for each validation.

Fig. 9. Attention heatmaps.

Example attention heatmaps from the ABMIL classifier using the ImageNet-pretrained ResNet50 and UNI foundation model features. (Upper) A typical difference between heatmaps with different diagnoses. (Lower) The most extreme qualitative difference found between heatmaps in the internal test set. In both examples, the UNI classification was correct (upper—MC, lower—CCC), and the ResNet50 classification was incorrect (upper—EC, lower—MC). These heatmaps are based on 256 × 256 pixel patches with 50% overlap at 10× apparent magnification, with visual differences in scale caused by the variable size of resection samples.