A survey of Transformer applications for histopathological image analysis: New developments and future directions

Abstract

Transformers have been widely used in many computer vision challenges and have shown the capability of producing better results than convolutional neural networks (CNNs). Taking advantage of capturing long-range contextual information and learning more complex relations in the image data, Transformers have been used and applied to histopathological image processing tasks. In this survey, we make an effort to present a thorough analysis of the uses of Transformers in histopathological image analysis, covering several topics, from the newly built Transformer models to unresolved challenges. To be more precise, we first begin by outlining the fundamental principles of the attention mechanism included in Transformer models and other key frameworks. Second, we analyze Transformer-based applications in the histopathological imaging domain and provide a thorough evaluation of more than 100 research publications across different downstream tasks to cover the most recent innovations, including survival analysis and prediction, segmentation, classification, detection, and representation. Within this survey work, we also compare the performance of CNN-based techniques to Transformers based on recently published papers, highlight major challenges, and provide interesting future research directions. Despite the outstanding performance of the Transformer-based architectures in a number of papers reviewed in this survey, we anticipate that further improvements and exploration of Transformers in the histopathological imaging domain are still required in the future. We hope that this survey paper will give readers in this field of study a thorough understanding of Transformer-based techniques in histopathological image analysis, and an up-to-date paper list summary will be provided at https://github.com/S-domain/Survey-Paper .

Keywords: CNN; Digital pathology; Histopathological imaging; Survival analysis; Transformer; Whole slide image.

Fig. 1

Some samples of histopathological images. a Whole slide images (WSIs). b Annotated PanNuke dataset from different tissue types for nuclei instance classification and segmentation

Fig. 2

Current transformer applications in histopathological image analysis, as surveyed in this research work

Fig. 3

A schematic demonstration of the self-attention mechanism

Fig. 4

A schematic demonstration of a standard transformer architecture

Fig. 5

A schematic diagram of a standard ViT model. Sequential image patches are used as the input, which is then processed with a transformer encoder and uses an MLP head module to generate a class prediction

Fig. 6

Transformer-based architectures for histopathological image analysis. The figure shows some of the existing approaches for different downstream tasks, including segmentation, survival analysis and prediction, representation, detection, and classification

Fig. 7

The chart a displays the statistics of the papers presented in this survey according to histopathological imaging problem settings. The rightmost figure b demonstrates consistent growth in recent development (from 2019 to July 2023)

Fig. 8

Some typical transformer U-shaped architectures

Fig. 9

Some examples of SOTA transformer architectures for histopathological image classification

Fig. 10

Some examples of SOTA transformer architectures for histopathological image segmentation

Fig. 11

Examples of segmentation results of popular Unet architecture [38] and transformer-based models (ATTransUNet [2], DS-TransUNet [15], MedT [90], Diao et al. [86], TransAttUnet  [87], and NST [89]) on different histopathological datasets

Fig. 12

Some examples of SOTA transformer architectures for histopathological image detection

Fig. 13

Some examples of SOTA transformer architectures for histopathological image survival analysis and prediction

Fig. 14

A schematic illustration of the HIPT [22] transformer architecture for histopathological image representation