Vision transformer architecture and applications in digital health: a tutorial and survey

Abstract

The vision transformer (ViT) is a state-of-the-art architecture for image recognition tasks that plays an important role in digital health applications. Medical images account for 90% of the data in digital medicine applications. This article discusses the core foundations of the ViT architecture and its digital health applications. These applications include image segmentation, classification, detection, prediction, reconstruction, synthesis, and telehealth such as report generation and security. This article also presents a roadmap for implementing the ViT in digital health systems and discusses its limitations and challenges.

Keywords: Artificial intelligence; Digital health; Medical imaging; Telehealth; Vision transformer.

Fig. 1

Transformer architecture [1]

Fig. 2

Encoder block in the transformer architecture [1]

Fig. 3

a Illustration of splitting ultrasound images into patches and flattening them in a linear sequence; b Image patch vectorization and linear projection; c Patch embedding in multidimensional space

Fig. 4

Positional encoding for the feature representations. Top: Sinusoidal representation for the positional encoding (P0-P3) at different indices and dimensions. Bottom: Vector representation for the positional encoding and feature embedding; P is the position encoding and E is the embedding vector

Fig. 5

MSA process. a MSA process with several attention layers in parallel; b Scaled dot product [8]. The diagram flows upwards from the bottom according to the direction of the arrow

Fig. 6

MLP

Fig. 7

Decoder and mask multihead attention block to produce the final image

Fig. 8

Distribution of medical imaging applications of the ViT according to the survey [33]

Fig. 9

Comparison of TransUNet and GT using output segmentation results of different organs: a GT (expert reference) and b TransUNet [10]

Fig. 10

Example of using the ViT for tumor classification in MRI images using TransMed [53]. The tumor is enclosed by the dashed circle indicated by the yellow arrow

Fig. 11

Examples of using ViT for surgical instruction prediction. Transformer prediction is based on the SIGT method [62]. GT is used as a reference for comparison and validation

Fig. 12

Top: Different reconstruction methods from T₁ weighted acquisition of the fast MRI using different methods. ZF is a traditional Fourier method [70]. LORKAS [71, 72], GAN_sub [73], SSDU [74], GAN_prior [75], and SAGAN [76] are generative adversarial network (GAN) reconstruction-based methods. SLATER is a ViT-based method [69]. Bottom: Reconstruction error map [69]

Fig. 13

Schematic of the components of the ViT in a telehealth ecosystem

Fig. 14

Examples of report generation from the input image using the ViT. a Sample of results by the IFCC algorithm [89] for report completeness and consistency; b Example of report generation results by the RTMIC algorithm [88]

Fig. 15

Illustration of data poisoning by an adversarial attack that fools learning-based models trained on medical image datasets

Fig. 16

Roadmap for ViT implementation

Fig. 17

Comparison between ViT and ResNet (BiT) architecture accuracies on different sizes of training data. The y-axis is the size of pretraining data in the ImageNet dataset. The x-axis is the accuracy selected from the top 1% of the selected five-shots of ImageNet. Results according to the study in ref. [1]

Fig. 18

Transformer typical architecture [8]

Fig. 19

Example of using Transformer architecture for image recognition [1]

Fig. 20

a Transformer layer diagram; b TransUnet architecture [10]

Fig. 21

Swin TransUnet architecture [11]

Fig. 22

Examples of global features that are used for mortality predictions are numbered from (112-139). The numbers in the table depicts the rank sore and each column represents a feature and its importance score by different methods on the horizontal line [109]. AutoInt [111], LSTM [112], TCN [113], Transformer [8], IMVLSTM [114] are the machine learning methodologies