The Depth Estimation and Visualization of Dermatological Lesions: Development and Usability Study

Abstract

Background: Thus far, considerable research has been focused on classifying a lesion as benign or malignant. However, there is a requirement for quick depth estimation of a lesion for the accurate clinical staging of the lesion. The lesion could be malignant and quickly grow beneath the skin. While biopsy slides provide clear information on lesion depth, it is an emerging domain to find quick and noninvasive methods to estimate depth, particularly based on 2D images.

Objective: This study proposes a novel methodology for the depth estimation and visualization of skin lesions. Current diagnostic methods are approximate in determining how much a lesion may have proliferated within the skin. Using color gradients and depth maps, this method will give us a definite estimate and visualization procedure for lesions and other skin issues. We aim to generate 3D holograms of the lesion depth such that dermatologists can better diagnose melanoma.

Methods: We started by performing classification using a convolutional neural network (CNN), followed by using explainable artificial intelligence to localize the image features responsible for the CNN output. We used the gradient class activation map approach to perform localization of the lesion from the rest of the image. We applied computer graphics for depth estimation and developing the 3D structure of the lesion. We used the depth from defocus method for depth estimation from single images and Gabor filters for volumetric representation of the depth map. Our novel method, called red spot analysis, measures the degree of infection based on how a conical hologram is constructed. We collaborated with a dermatologist to analyze the 3D hologram output and received feedback on how this method can be introduced to clinical implementation.

Results: The neural model plus the explainable artificial intelligence algorithm achieved an accuracy of 86% in classifying the lesions correctly as benign or malignant. For the entire pipeline, we mapped the benign and malignant cases to their conical representations. We received exceedingly positive feedback while pitching this idea at the King Edward Memorial Institute in India. Dermatologists considered this a potentially useful tool in the depth estimation of lesions. We received a number of ideas for evaluating the technique before it can be introduced to the clinical scene.

Conclusions: When we map the CNN outputs (benign or malignant) to the corresponding hologram, we observe that a malignant lesion has a higher concentration of red spots (infection) in the upper and deeper portions of the skin, and that the malignant cases have deeper conical sections when compared with the benign cases. This proves that the qualitative results map with the initial classification performed by the neural model. The positive feedback provided by the dermatologist suggests that the qualitative conclusion of the method is sufficient.

Keywords: ML; MR; XAI; artificial intelligence; computer graphics; computer vision; explainable AI; machine learning; mixed reality; neural networks; red spot analysis; visualization.

Figure 1

Application of the GradCAM algorithm within the model. Conv2D: Convolutional 2-Dimensional; GradCAM: gradient class activation map.

Figure 2

The figure represents the evaluation metrics of the model. The confusion matrix gives us the TP, FP, FN, and TN values, allowing us to calculate the other metrics. The AUC-ROC curve represents the TP-FP trade-off, and the AUC-PR curve represents the precision-recall trade-off. AUC-PR: area under the precision-recall curve; AUC-ROC: area under the receiver operating characteristics curve; FN: false negative; FP: false positive; MCC: Matthews correlation coefficient; TN: true negative; TP: true positive.

Figure 3

The figure represents the output of the testing pipeline. We have the original image, the image after localization, the depth map, the 3D representation of the map, and the conical representation of the hologram (left-to-right).