Integrating Foundation Model Features into Graph Neural Network and Fusing Predictions with Standard Fine-Tuned Models for Histology Image Classification

Abstract

Histopathological image classification using computational methods such as fine-tuned convolutional neural networks (CNNs) has gained significant attention in recent years. Graph neural networks (GNNs) have also emerged as strong alternatives, often employing CNNs or vision transformers (ViTs) as node feature extractors. However, as these models are usually pre-trained on small-scale natural image datasets, their performance in histopathology tasks can be limited. The introduction of foundation models trained on large-scale histopathological data now enables more effective feature extraction for GNNs. In this work, we integrate recently developed foundation models as feature extractors within a lightweight GNN and compare their performance with standard fine-tuned CNN and ViT models. Furthermore, we explore a prediction fusion approach that combines the outputs of the best-performing GNN and fine-tuned model to evaluate the benefits of complementary representations. Results demonstrate that GNNs utilizing foundation model features outperform those trained with CNN or ViT features and achieve performance comparable to standard fine-tuned CNN and ViT models. The highest overall performance is obtained with the proposed prediction fusion strategy. Evaluated on three publicly available datasets, the best fusion achieved F1-scores of 98.04%, 96.51%, and 98.28%, and balanced accuracies of 98.03%, 96.50%, and 97.50% on PanNuke, BACH, and BreakHis, respectively.

Keywords: computational pathology; deep learning; foundation model; graph neural network; image classification; medical image analysis.

Figure 1

Example images from the PanNuke dataset (first row), the BACH dataset (second row) and the BreakHis dataset (third row).

Figure 2

General workflow of the proposed approach.

Figure 3

Visual comparison of the best-performing methods on the PanNuke, BACH, and BreakHis datasets. For each dataset, the performance of the best models from each part of the corresponding tables (Table 2, Table 3 and Table 4) is shown.

Figure 4

Attention heatmaps generated using the Grad-CAM method for two example images are shown for the GNN-UNI2 and GNN-ViT models. In both cases, GNN-UNI2 correctly classified the images, whereas GNN-ViT produced incorrect predictions. As illustrated, GNN-UNI2 focuses more strongly on clinically relevant regions of the tissue compared to GNN-ViT.