Predicting the clinical management of skin lesions using deep learning

Abstract

Automated machine learning approaches to skin lesion diagnosis from images are approaching dermatologist-level performance. However, current machine learning approaches that suggest management decisions rely on predicting the underlying skin condition to infer a management decision without considering the variability of management decisions that may exist within a single condition. We present the first work to explore image-based prediction of clinical management decisions directly without explicitly predicting the diagnosis. In particular, we use clinical and dermoscopic images of skin lesions along with patient metadata from the Interactive Atlas of Dermoscopy dataset (1011 cases; 20 disease labels; 3 management decisions) and demonstrate that predicting management labels directly is more accurate than predicting the diagnosis and then inferring the management decision ([Formula: see text] and [Formula: see text] improvement in overall accuracy and AUROC respectively), statistically significant at [Formula: see text]. Directly predicting management decisions also considerably reduces the over-excision rate as compared to management decisions inferred from diagnosis predictions (24.56% fewer cases wrongly predicted to be excised). Furthermore, we show that training a model to also simultaneously predict the seven-point criteria and the diagnosis of skin lesions yields an even higher accuracy (improvements of [Formula: see text] and [Formula: see text] in overall accuracy and AUROC respectively) of management predictions. Finally, we demonstrate our model's generalizability by evaluating on the publicly available MClass-D dataset and show that our model agrees with the clinical management recommendations of 157 dermatologists as much as they agree amongst each other.

Figure 1

An overview of the three prediction models. All the models take the clinical and the dermoscopic images of the skin lesion and the patient metadata as input. Note that we also perform an input ablation study (A multi-task prediction model section; Table 4). (a) The first model predicts the lesion diagnosis probabilities, ${\mathrm{DIAG}}_{\mathrm{pred}}$. (b) The second model predicts the management decision probabilities, ${\mathrm{MGMT}}_{\mathrm{pred}}$. (c) The third is a multi-task model and predicts the seven-point criteria ($\mathrm{Criterion}{\{1,2,\dots ,7\}}_{\mathrm{pred},\mathrm{multi}}$) in addition to ${\mathrm{DIAG}}_{\mathrm{pred},\mathrm{multi}}$ and ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{multi}}$. The argmax operation assigns 1 to the most likely label and 0 to all others. For (a), ${\mathrm{DIAG}}_{\mathrm{pred}}$ diagnosis is used to arrive at a management decision either using (a1) binary labeling, ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{binary}}$, or (a2) prior based inference, ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{all}}$. Similarly, the outputs of (b) can be used to directly predict a management decision using either (b1) binary labeling, ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{binary}}$, or (b2) all the labels, ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{all}}$. As explained in the text, the diagnosis labels are basal cell carcinoma (BCC), nevus (NEV), melanoma (MEL), seborrheic keratosis (SK), and others (MISC), and the management decision labels are ‘clinical follow up’ (CLNC), ‘excision’ (EXC), and ‘no further examination’ (NONE). In the case of binary management decisions, we predict whether a lesion should be excised (EXC) or not (NOEXC).

Figure 2

Quantitative evaluation of the ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{all}}$ and ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{all}}$ predictions. (a) Violin plots of the distance measures of the probabilistic predictions show that the ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{all}}$ predictions are closer (statistically significant) to the target labels for test data. (b, c) ROC curves and (d, e) confusion matrices of ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{all}}$ and ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{all}}$ respectively along with cell-wise diagnosis breakdown. Note that ${\mathrm{MGMT}}_{\mathrm{infr},\mathrm{all}}$ has a tendency to over-excise lesions.

Figure 3

Evaluating the multi-modal multi-task model. (a) ROC curve and (b) precision-recall curve for the management prediction task. Confusion matrices for (c) the management prediction task and (d) the diagnosis prediction task along with the diagnosis-wise breakdown for the management labels.

Figure 4

Evaluating the statistical significance of each input data modality’s contribution in improving the management decision prediction ${\mathrm{MGMT}}_{\mathrm{pred},\mathrm{multi}}$. ‘C’, ‘D’, and ‘M’ refer to clinical image, dermoscopic image, and patient metadata respectively, and the row and the column names refer to the experiments in the ablation study presented in Table 4. For each pair of experiments (i) and (j), the cell (i, j) contains the p-value corresponding to the McNemar’s test performed on the corresponding pair of predictions.

Figure 5

Evaluating the multi-task model on the MClass-D dataset. (a) Confusion matrices and (b) ROC curves for ${\mathrm{MGMT}}_{\mathrm{pred}}$ and ${\mathrm{MGMT}}_{\mathrm{infr}}$ predictions with both ${\mathrm{MGMT}}_{\mathrm{GT},\mathrm{agg}}$ and ${\mathrm{MGMT}}_{\mathrm{GT},\mathrm{true}}$ as target clinical management labels.

Figure 6

A breakdown of the inputs, outputs, loss functions, and architecture of the three prediction models. Global average pooled feature responses from the clinical and the dermoscopic images are extracted and concatenated (denoted by the plus symbol) with one-hot encoded patient meta-data, and the three models are trained with $L_{\mathrm{DIAG}}$, $L_{\mathrm{MGMT}}$, and $L_{\mathrm{multi}}$ respectively. The first model predicts the diagnosis labels (${\mathrm{DIAG}}_{\mathrm{pred}}$) which are then used along with the management priors to obtain inferred management decisions (${\mathrm{MGMT}}_{\mathrm{infr}}$), whereas the second model predicts the management decisions directly (${\mathrm{MGMT}}_{\mathrm{pred}}$). Finally, the last model is a multi-task one and is trained to predict the seven-point criteria, the diagnosis, and the management (outputs enclosed in the dashed box).