Transformer attention fusion for fine grained medical image classification

Abstract

Fine-grained visual classification is fundamental for medical image applications because it detects minor lesions. Diabetic retinopathy (DR) is a preventable cause of blindness, which requires exact and timely diagnosis to prevent vision damage. The challenges automated DR classification systems face include irregular lesions, uneven distributions between image classes, and inconsistent image quality that reduces diagnostic accuracy during early detection stages. Our solution to these problems includes MSCAS-Net (Multi-Scale Cross and Self-Attention Network), which uses the Swin Transformer as the backbone. It extracts features at three different resolutions (12 × 12, 24 × 24, 48 × 48), allowing it to detect subtle local features and global elements. This model uses self-attention mechanics to improve spatial connections between single scales and cross-attention to automatically match feature patterns across multiple scales, thereby developing a comprehensive information structure. The model becomes better at detecting significant lesions because of its dual mechanism, which focuses on both attention points. MSCAS-Net displays the best performance on APTOS and DDR and IDRID benchmarks by reaching accuracy levels of 93.8%, 89.80% and 86.70%, respectively. Through its algorithm, the model solves problems with imbalanced datasets and inconsistent image quality without needing data augmentation because it learns stable features. MSCAS-Net demonstrates a breakthrough in automated DR diagnostics since it combines high diagnostic precision with interpretable abilities to become an efficient AI-powered clinical decision support system. The presented research demonstrates how fine-grained visual classification methods benefit detecting and treating DR during its early stages.

Keywords: Attention mechanism; Deep learning; Diabetic retinopathy classification; Fine-grained visual classification; Medical images; Multi-scale feature extraction.

Fig. 1

MSCAS-Net (multi-scale cross and self-attention network) architecture.

Fig. 2

Self-attention block.

Fig. 3

Cross attention block.

Fig. 4

Training and validation accuracy curves for APTOS dataset.

Fig. 5

Training and validation loss curves for APTOS dataset.

Fig. 6

Training and validation accuracy curves for DDR dataset.

Fig. 7

Training and validation loss curves for DDR dataset.

Fig. 8

Training and validation accuracy curves for IDRID dataset.

Fig. 9

Training and validation loss curves for IDRID dataset.

Fig. 10

A visual example of heatmaps generated using this model on three datasets (a) APTOS (b) DDR and (c) IDRID.