DSP-KD: Dual-Stage Progressive Knowledge Distillation for Skin Disease Classification

Abstract

The increasing global demand for skin disease diagnostics emphasizes the urgent need for advancements in AI-assisted diagnostic technologies for dermatoscopic images. In current practical medical systems, the primary challenge is balancing lightweight models with accurate image analysis to address constraints like limited storage and computational costs. While knowledge distillation methods hold immense potential in healthcare applications, related research on multi-class skin disease tasks is scarce. To bridge this gap, our study introduces an enhanced multi-source knowledge fusion distillation framework, termed DSP-KD, which improves knowledge transfer in a dual-stage progressive distillation approach to maximize mutual information between teacher and student representations. The experimental results highlight the superior performance of our distilled ShuffleNetV2 on both the ISIC2019 dataset and our private skin disorders dataset. Compared to other state-of-the-art distillation methods using diverse knowledge sources, the DSP-KD demonstrates remarkable effectiveness with a smaller computational burden.

Keywords: ISIC2019; diverse knowledge; knowledge distillation; skin disease classification.

Figure 1

The two knowledge sources obtained from the teacher network.

Figure 2

The overall structure of the proposed DSP-KD framework.

Figure 3

The structure of the proposed HF attention map mechanism.

Figure 4

Illustration of the generation block.

Figure 5

Demonstration of the effects of data augmentation techniques.

Figure 6

Sample images along with the detailed introduction about the ISIC2019 dataset.

Figure 7

Sample images along with the detailed introduction about the PSD-Dataset.

Figure 8

Confusion matrix.

Figure 9

The training process of the three model pairs (Res2Net101/ShuffleNetV2, ConvNext/ShuffleNetV2, EfficientNet-B4/ShuffleNetV2) under our DSP-KD framework.

Figure 10

The confusion matrix of the training results of our proposed DSP-KD method.