A generalised vision transformer-based self-supervised model for diagnosing and grading prostate cancer using histological images

Abstract

Background: Gleason grading remains the gold standard for prostate cancer histological classification and prognosis, yet its subjectivity leads to grade variability between pathologists, potentially impacting clinical decision-making. Herein, we trained and validated a generalised AI-driven system for diagnosing prostate cancer using diverse datasets from tissue microarray (TMA) core and whole slide images (WSIs) with Haematoxylin and Eosin staining.

Methods: We analysed eight prostate cancer datasets, which included 12,711 histological images from 3648 patients, incorporating TMA core images and WSIs. The Macenko method was used to normalise colours for consistency across diverse images. Subsequently, we trained a multi-resolution (5x, 10x, 20x, and 40x) binary classifier to identify benign and malignant tissue. We then implemented a multi-class classifier for Gleason patterns (GP) sub-categorisation from malignant tissue. Finally, the models were externally validated on 11,132 histology images from 2176 patients to determine the International Society of Urological Pathology (ISUP) grade. Models were assessed using various classification metrics, and the agreement between the model's predictions and the ground truth was quantified using the quadratic weighted Cohen's Kappa (κ) score.

Results: Our multi-resolution binary classifier demonstrated robust performance in distinguishing malignant from benign tissue with κ scores of 0.967 on internal validation. The model achieved κ scores ranging from 0.876 to 0.995 across four unseen testing datasets. The multi-class classifier also distinguished GP3, GP4, and GPs with an overall κ score of 0.841. This model was further tested across four datasets, obtaining κ scores ranging from 0.774 to 0.888. The models' performance was compared against an independent pathologist's annotation on an external dataset, achieving a κ score of 0.752 for four classes.

Conclusion: The self-supervised ViT-based model effectively diagnoses and grades prostate cancer using histological images, distinguishing benign and malignant tissues and classifying malignancies by aggressiveness. External validation highlights its robustness and clinical applicability in digital pathology.

Fig. 1. Shows patches and corresponding attention maps for both benign and malignant cases.

Three randomly predicted patches from each class are shown, with actual and predicted labels and the model’s prediction confidence for each testing dataset.

Fig. 2. Visualisation of attention maps for Gleason patterns 3, 4, and 5.

Two randomly selected patches from each class display actual and predicted labels with the model’s confidence for each testing dataset.

Fig. 3. Gleason grading at the slide level.

The developed models detected and highlighted Gleason patterns (3, 4, and 5) and established ISUP grade groups from diverse histology images across three external datasets (PANDA, NADT-Prostate, and PROSTATE-MRI).