A deep-learning algorithm to classify skin lesions from mpox virus infection

Abstract

Undetected infection and delayed isolation of infected individuals are key factors driving the monkeypox virus (now termed mpox virus or MPXV) outbreak. To enable earlier detection of MPXV infection, we developed an image-based deep convolutional neural network (named MPXV-CNN) for the identification of the characteristic skin lesions caused by MPXV. We assembled a dataset of 139,198 skin lesion images, split into training/validation and testing cohorts, comprising non-MPXV images (n = 138,522) from eight dermatological repositories and MPXV images (n = 676) from the scientific literature, news articles, social media and a prospective cohort of the Stanford University Medical Center (n = 63 images from 12 patients, all male). In the validation and testing cohorts, the sensitivity of the MPXV-CNN was 0.83 and 0.91, the specificity was 0.965 and 0.898 and the area under the curve was 0.967 and 0.966, respectively. In the prospective cohort, the sensitivity was 0.89. The classification performance of the MPXV-CNN was robust across various skin tones and body regions. To facilitate the usage of the algorithm, we developed a web-based app by which the MPXV-CNN can be accessed for patient guidance. The capability of the MPXV-CNN for identifying MPXV lesions has the potential to aid in MPXV outbreak mitigation.

Fig. 1. Flow diagram for the MPXV and non-MPXV image datasets.

The flow diagram showed the identification and screening procedures of images to create the MPXV and non-MPXV datasets. MPXV images were collected from publications of the scientific literature, encyclopedia articles, new articles, social media and a prospective cohort of patients from the Stanford University Medical Center, while non-MPXV images originated from eight repositories and datasets.

Fig. 2. Performance diagrams of the MPXV-CNN for the validation and testing cohorts.

a, ROC curve derived from repeated fivefold cross-validation on the validation cohort (AUC = 0.967 ± 0.003). b, Confusion matrix on the testing cohort showing the ratios of TPs (0.91), TNs (0.898), FPs (0.102) and FNs (0.09). c, ROC curve of the testing cohort that included MPXV skin lesions and either acute non-MPXV skin lesions (AUC = 0.962), chronic non-MPXV skin lesions (AUC = 0.967) or all non-MPXV skin lesions (AUC = 0.966). FPs, false positives; FNs, false negatives; ROC, receiver operating characteristic; TPs, true positives; TNs, true negatives.

Fig. 3. SHAP analysis of the MPXV-CNN.

Photographic images of MPXV skin lesions (top) are shown with the corresponding SHAP analysis (bottom) overlaid on the original image to highlight the discriminative image regions used for detection (a–g). The MPXV lesions shown represent different stages as follows: early-stage vesicle (a), small pustule (b), umbilicated pustule (c), papule with central necrosis (d), hand with one ulcerated skin lesion (e), pubic region with multiple ulcerated skin lesions (f) and late-stage crusted plaques (g). Positive SHAP values, shown in red, indicated areas of the image that contributed to the prediction of MPXV skin lesion, whereas negative SHAP values, shown in blue, indicated areas that detracted from the prediction. All MPXV lesions shown in a–g were part of the testing dataset and were classified correctly by the MPXV-CNN. Photo credit (a–g): UK Health Security Agency, licensed under the Open Government License 3.0.

Fig. 4. Screenshots of PoxApp.

a, Screenshots of the start screen are shown. b, Question regarding the presence of new lesions. c, Prompt for taking a photograph of the skin lesion. d, Question regarding further symptoms. e, Question regarding close contacts with infected individuals. f, A personalized recommendation computed from the information provided and the MPXV-CNN classification of the skin lesion image.

Fig. 5. Components of the PRS for MPXV patient guidance.

a, Simplified decision tree for MPXV infection risk stratification derived from WHO case definitions with the addition of an AI-assisted case definition based on predictions of the MPXV-CNN. An IDE was used to create and update the survey for risk stratification (boxes) based on these questions (rhombuses), logical expressions (arrows) and the MPXV-CNN (rhombus with a brain and AI model). An API distributed the most up-to-date survey, logical expressions and MPXV-CNN to web-based apps. b, The web-based app ‘PoxApp’ implemented the PRS for end users allowing them to answer surveys and take photos of their skin lesions and get personalized recommendations, such as MPXV testing or vaccination. c, Component for voluntary data donation with an API to collect, anonymize and store data in a central database. d, New evolving models with higher sensitivity and specificity could potentially be created based on new user data. API, application programming interface.

Extended Data Fig. 1. Subgroup analysis of the sensitivity in the testing cohort.

The observed sensitivity was high in the prospective cohort (0.89) with patients from the Stanford University Medical Center and in other MPXV images (0.92). MPXV, mpox virus; n, Number of available images per testing cohort.

Extended Data Fig. 2. False Positive Rates in 7 non-MPXV image repositories and datasets of the testing cohort.

n, Number of available images per image repository.

Extended Data Fig. 3. True Positive Rates by duration of presence of the MPXV skin lesion in the testing cohort.

n, Number of available images per group.

Extended Data Fig. 4. True Positive Rates by number of visible MPXV skin lesions N in the testing cohort.

n, Number of available images per group; N, Number of visible MPXV skin lesions in the image.

Extended Data Fig. 5. Specificity for classifying MPXV skin lesions versus acute and chronic non-MPXV skin diseases.

n, Number of available images per group.

Extended Data Fig. 6. Top 30 False Positive Rates of acute diagnoses in the testing cohort with at least 50 available images.

The full list of diagnoses and False Positive Rates can be found in Supplementary Tables 1–11. n, Number of available images per diagnosis.

Extended Data Fig. 7. Top 30 False Positive Rates of chronic diagnoses in the testing cohort with at least 50 available images.

The full list of diagnoses and False Positive Rates can be found in Supplementary Tables 1–11. n, Number of available images per diagnosis.

Extended Data Fig. 8. True Positive Rates by body region in the testing cohort.

n, Number of available images per body region.

Extended Data Fig. 9. True Positive Rates by skin tone (Fitzpatrick Type) in the testing cohort.

n, Number of available images per group.

Extended Data Fig. 10. False Positive Rates by skin tone (Fitzpatrick Type) of non-MPXV images of the Fitzpatrick 17k dataset.

The highest False Positive Rates could be observed in skin tone Fitzpatrick Types V and VI. n, Number of available images per group.