Boosting Skin Cancer Classification: A Multi-Scale Attention and Ensemble Approach with Vision Transformers

Abstract

Skin cancer is a significant global health concern, with melanoma being the most dangerous form, responsible for the majority of skin cancer-related deaths. Early detection of skin cancer is critical, as it can drastically improve survival rates. While deep learning models have achieved impressive results in skin cancer classification, there remain challenges in accurately distinguishing between benign and malignant lesions. In this study, we introduce a novel multi-scale attention-based performance booster inspired by the Vision Transformer (ViT) architecture, which enhances the accuracy of both ViT and convolutional neural network (CNN) models. By leveraging attention maps to identify discriminative regions within skin lesion images, our method improves the models' focus on diagnostically relevant areas. Additionally, we employ ensemble learning techniques to combine the outputs of several deep learning models using majority voting. Our skin cancer classifier, consisting of ViT and EfficientNet models, achieved a classification accuracy of 95.05% on the ISIC2018 dataset, outperforming individual models. The results demonstrate the effectiveness of integrating attention-based multi-scale learning and ensemble methods in skin cancer classification.

Keywords: deep learning; image processing; neural networks; skin cancer classification; transformer.

Figure 1

A multi-scale attention and ensemble approach with Vision Transformers.

Figure 2

ISIC2018 image set samples.

Figure 3

Sensitivity/precision comparison for each type trained with/without oversampling [24].

Figure 4

Attention maps generated in ViT for skin tumors.

Figure 5

The attention map assists in identifying the discriminate region of the tumor.

Figure 6

ViT-based multi-scale performance booster attached to a deep learning model.

Figure 7

Attention map focusing on background objects for low-contract lesion.

Figure 8

Some misclassified melanoma images.