Diagnostic accuracy of content-based dermatoscopic image retrieval with deep classification features

Abstract

Background: Automated classification of medical images through neural networks can reach high accuracy rates but lacks interpretability.

Objectives: To compare the diagnostic accuracy obtained by using content-based image retrieval (CBIR) to retrieve visually similar dermatoscopic images with corresponding disease labels against predictions made by a neural network.

Methods: A neural network was trained to predict disease classes on dermatoscopic images from three retrospectively collected image datasets containing 888, 2750 and 16 691 images, respectively. Diagnosis predictions were made based on the most commonly occurring diagnosis in visually similar images, or based on the top-1 class prediction of the softmax output from the network. Outcome measures were area under the receiver operating characteristic curve (AUC) for predicting a malignant lesion, multiclass-accuracy and mean average precision (mAP), measured on unseen test images of the corresponding dataset.

Results: In all three datasets the skin cancer predictions from CBIR (evaluating the 16 most similar images) showed AUC values similar to softmax predictions (0·842, 0·806 and 0·852 vs. 0·830, 0·810 and 0·847, respectively; P > 0·99 for all). Similarly, the multiclass-accuracy of CBIR was comparable with softmax predictions. Compared with softmax predictions, networks trained for detecting only three classes performed better on a dataset with eight classes when using CBIR (mAP 0·184 vs. 0·368 and 0·198 vs. 0·403, respectively).

Conclusions: Presenting visually similar images based on features from a neural network shows comparable accuracy with the softmax probability-based diagnoses of convolutional neural networks. CBIR may be more helpful than a softmax classifier in improving diagnostic accuracy of clinicians in a routine clinical setting.

Figure 1

Positive examples of three query images (first column) and corresponding most similar images as found by content‐based image retrieval (CBIR). The results show similar dermatoscopic patterns that in the majority correspond to the correct diagnosis. MEL, melanoma; SCC, squamous cell carcinoma; BCC, basal cell carcinoma.

Figure 2

Measured visual similarity (cosine similarity) of images with the same diagnoses (blue) compared with others (red) in a dataset. Images of the same diagnoses are significantly rated higher in almost any subgroup, showing automated measurements of visual similarity can differentiate between diagnoses within a retrieval dataset. Lines are drawn between values for a single query image, and rows denote the dataset used for training, queries and image retrieval. Comparing differences was performed with a paired t‐test or a paired Wilcoxon signed rank test (W). ISIC2017, International Skin Imaging Collaboration, bcc, basal cell carcinoma; bkl, seborrhoeic keratoses; df, dermatofibromas; inflammatory, inflammatory lesions including dermatitis, lichen sclerosus, porokeratosis, rosacea, psoriasis, lupus erythematosus, bullous pemphigoid, lichen planus, granulomatous processes and artefacts mel, melanoma; scc, squamous cell carcinoma. NS, nonsignificant: P > 0·05, **P < 0·01, ***P < 0·001.

Figure 3

Frequency of correct specific diagnoses (accuracy) made within each dataset by either softmax‐based predictions (red) or content‐based image retrieval (CBIR) with a different number of retrieved similar images (black). Retrieval of a few images is already performing better in the three‐class datasets (EDRA, ISIC2017), whereas in the eight‐class (PRIV) dataset it takes over 20 images to approximate softmax‐based accuracy. ISIC2017, International Skin Imaging Collaboration.

Figure 4

Receiver operating characteristic curve for detecting melanoma when retrieving 16 similar images with content‐based image retrieval (CBIR) (grey), showing different thresholds of needed malignant retrieval images (‘predict melanoma when x of 16 retrieved images are melanomas’), in addition to softmax‐based probabilities (red). Network training‐, query‐ and retrieval‐images are from EDRA. AUC, area under the receiver operating characteristic curve.

Figure 5

Mean average precision (mAP) of a ResNet‐50 network trained on EDRA dataset images. Predictions were made either through softmax probabilities (red line) or class‐frequencies of content‐based image retrieval (CBIR) (black). Softmax predictions perform worst on predicting PRIV dataset images, as the networks are not able to predict five of the eight classes in any case (first two columns, bottom row). CBIR retrieving from EDRA and ISIC2017 suffers from the same shortcomings, but was able to predict better when using PRIV‐source retrieval images (bottom right). In general, CBIR performs best when using retrieval images from the same source as the test images (descending diagonal), and here performed better on new data than softmax predictions. Re‐training the network on those new‐source images (blue) in turn outperformed CBIR again. ISIC2017, International Skin Imaging Collaboration.

Figure 6

Mean cosine similarities of PRIV retrieval images with the same (blue) or different (red) diagnosis for the corresponding PRIV query images. Cosine similarity is calculated by feature extraction via ResNet‐50 networks trained for classification on different training datasets (rows). Compared with the PRIV‐trained network, those trained on different sources (row EDRA and ISIC2017) transfer their ability to distinguish specific diagnoses through visual similarity except for seborrhoeic keratoses (bkl) cases. Lines are drawn between values for the same query image. W, paired Wilcoxon signed‐rank test was used instead of paired t‐test; ISIC2017, International Skin Imaging Collaboration; bcc, basal cell carcinoma; df, dermatofibromas; mel, melanoma; scc, squamous cell carcinoma. NS, nonsignificant: P > 0·05, **P < 0·01, ***P < 0·001; grey indicators denote nonadjusted P‐values as these comparisons were omitted during correction for multiple testing (see statistics section).