Prospective validation of dermoscopy-based open-source artificial intelligence for melanoma diagnosis (PROVE-AI study)

Abstract

The use of artificial intelligence (AI) has the potential to improve the assessment of lesions suspicious of melanoma, but few clinical studies have been conducted. We validated the accuracy of an open-source, non-commercial AI algorithm for melanoma diagnosis and assessed its potential impact on dermatologist decision-making. We conducted a prospective, observational clinical study to assess the diagnostic accuracy of the AI algorithm (ADAE) in predicting melanoma from dermoscopy skin lesion images. The primary aim was to assess the reliability of ADAE's sensitivity at a predefined threshold of 95%. Patients who had consented for a skin biopsy to exclude melanoma were eligible. Dermatologists also estimated the probability of melanoma and indicated management choices before and after real-time exposure to ADAE scores. All lesions underwent biopsy. Four hundred thirty-five participants were enrolled and contributed 603 lesions (95 melanomas). Participants had a mean age of 59 years, 54% were female, and 96% were White individuals. At the predetermined 95% sensitivity threshold, ADAE had a sensitivity of 96.8% (95% CI: 91.1-98.9%) and specificity of 37.4% (95% CI: 33.3-41.7%). The dermatologists' ability to assess melanoma risk significantly improved after ADAE exposure (AUC 0.7798 vs. 0.8161, p = 0.042). Post-ADAE dermatologist decisions also had equivalent or higher net benefit compared to biopsying all lesions. We validated the accuracy of an open-source melanoma AI algorithm and showed its theoretical potential for improving dermatology experts' ability to evaluate lesions suspicious of melanoma. Larger randomized trials are needed to fully evaluate the potential of adopting this AI algorithm into clinical workflows.

Fig. 1. Concordance of expected sensitivity with observed sensitivity.

Observed (vertical axis) sensitivity (blue) and specificity (red) at various expected sensitivity thresholds (horizontal axis). A perfect concordance between observed and expected sensitivity is represented by the solid black line. Deviations of the blue line below the black line (observed sensitivity less than expected sensitivity) and above black line (observed sensitivity greater than expected sensitivity) suggest loss of concordance. At the predetermined 95% sensitivity threshold (the study’s prespecified primary endpoint), ADAE had a sensitivity of 96.8% (95% CI: 91.1–98.9%) and specificity of 37.4% (95% CI: 33.3–41.7%).

Fig. 2. ADAE score distribution by treating dermatologist and pathologic diagnosis.

ADAE scores (y-axis) for individual study lesions (yellow dots and blue triangles) are shown for melanomas (left panel) and non-melanomas (right panel), stratified by study dermatologist (x-axis). Higher ADAE scores (more likely melanoma) are closer to 0 and lower ADAE scores (less likely melanoma) are increasingly negative. The horizontal line is the prespecified 95% sensitivity threshold. Yellows dots represent lesions with an unequivocal benign or malignant diagnosis. Blue triangles represent lesions with a borderline pathology that was adjudicated to be melanoma or non-melanoma. The center line within each boxplot represents the median. The lower and upper hinges of the box represent the first and third quartiles (Q1 and Q3). The upper end of each whisker represents the more extreme value between the largest observed value and Q3 + 1.5 * IQR and the lower end of each whisker represents the more extreme value between the smallest observed value and Q1−1.5 * IQR, where IQR is the interquartile range.

Fig. 3. Decision curve plotting decrease in avoidable interventions against threshold probability.

The binary management decision is the dermatologist biopsy decision (yes/no) after exposure to AI results. The risk probability estimate is the dermatologist’s predicted melanoma probability (0–100%) after exposure to AI results. The default strategy (biopsy all lesions) is represented by the x-axis (net avoidable interventions of 0). For example, at a threshold probability of 5% (meaning that harm of missing 1 melanoma is equivalent to the harm of 19 unnecessary benign skin biopsies), exposing dermatologists to ADAE results would theoretically be the equivalent of a strategy that reduced the number of unnecessary biopsies by about 15–20 per 100 without missing biopsy for any patients with melanoma. All 22 histopathologic keratinocyte carcinomas were excluded from decision curve analyses because they are not viewed as equivalent to benign skin lesions.

Fig. 4. ADAE web-app interface.

Example of image upload results. a Dermoscopy image chosen by the dermatologist. b The ADAE score is shown as the vertical black line, calculated as the log-average of the 18 models (90 folds). Red (melanoma) and blue (benign lesions) dots were from the 2020 SIIM-ISIC Melanoma Challenge, with scores calculated as the average of the 18 models (90 folds). Higher scores (more likely melanoma) were to the right and lower scores (more likely benign) were to the left. The horizontal red (melanoma) and blue (benign lesions) lines show the distributions of ADAE scores for these diagnostic classes. Scores were spline transformed so the 95% sensitivity of the raw average of the 18 models was displayed in the center (gray vertical line). Users could adjust the interface to visualize different sensitivity thresholds (95%, 90%, 85%, and 80%) and different data test sets (red and blue dots) from the SIIM-ISIC Melanoma Challenge (MSK only vs. all 6 sites). c Saliency map showing the spatial support for melanoma prediction with yellow color indicating more likely melanoma and blue color indicating more likely not melanoma.