Reliable Detection of Eczema Areas for Fully Automated Assessment of Eczema Severity from Digital Camera Images

Abstract

Assessing the severity of eczema in clinical research requires face-to-face skin examination by trained staff. Such approaches are resource-intensive for participants and staff, challenging during pandemics, and prone to inter- and intra-observer variation. Computer vision algorithms have been proposed to automate the assessment of eczema severity using digital camera images. However, they often require human intervention to detect eczema lesions and cannot automatically assess eczema severity from real-world images in an end-to-end pipeline. We developed a model to detect eczema lesions from images using data augmentation and pixel-level segmentation of eczema lesions on 1,345 images provided by dermatologists. We evaluated the quality of the obtained segmentation compared with that of the clinicians, the robustness to varying imaging conditions encountered in real-life images, such as lighting, focus, and blur, and the performance of downstream severity prediction when using the detected eczema lesions. The quality and robustness of eczema lesion detection increased by approximately 25% and 40%, respectively, compared with that of our previous eczema detection model. The performance of the downstream severity prediction remained unchanged. Use of skin segmentation as an alternative to eczema segmentation that requires specialist labeling showed the performance on par with when eczema segmentation is used.

Figure 1

Overview of the EczemaNet1 pipeline. Reproduced with permission from Pan et al. (2020). The Region-of-Interest (RoI) detection model generates AD crops of the input image that contain AD regions. The severity prediction model makes probabilistic predictions of seven disease signs in each crop. The AD severity scores for each disease sign are integrated to give the regional severity scores for the whole image. AD, atopic dermatitis; EASI, Eczema Area and Severity Index; SASSAD, Six Area, Six Sign Atopic Dermatitis; RoI, region of interest.

Figure 2

Illustration of pixel-level segmentation masks. (a) A sample image and (b) the corresponding pixel-level segmentation masks of background (black), non-AD skin (gray), and AD skin (white). AD, atopic dermatitis.

Figure 3

Overview of the RoI detection part of EczemaNet2. (a) U-Net pixel-level segmentation for skin or AD and (b) postprocessing steps to produce crops that are inputs for the subsequent severity prediction model. Merging non-AD skin pixels with AD pixels is applied when both AD and skin segmentation are available. AD, atopic dermatitis; RoI, region of interest.

Figure 4

Quality and robustness of RoI detection. (a) Quality measured by F1 score and precision (mean ± SE; the higher, the better) on the test set of 271 photographs. (b) Robustness of RoI detection against image perturbations measured by IoU between predictions made from unperturbed and perturbed images (mean ± SE; the higher, the better) on the test set of 271 photographs, for different perturbations (Gaussian blurring, Gaussian noise, brightness change, and combinations). (c) Example AD segmentation of unperturbed and perturbed images by the RoI models trained with data augmentation (AD+Skin+DA) and with the nonaugmented dataset (AD+Skin). AD+Skin+DA generates consistent masks even in the presence of image perturbations but not AD+Skin. AD, atopic dermatitis; DA, data augmentation; IoU, intersection over union; RoI, region of interest; SE, standard error.

Figure 5

Accuracy of severity prediction evaluated with RMSE. Points show the mean $\pm$ SE over 10-fold cross-validation with the test set of 136 photographs for EASI (in [0, 12]), TISS (in [0, 9]), and SASSAD (in [0, 18]). EASI, Eczema Area and Severity Index; RMSE, root mean square error; SASSAD, Six Area, Six Sign Atopic Dermatitis; SE, standard error; TISS, Three-Item Severity score.

Figure 6

Performance of EczemaNet2 on skin of color subpopulation. Points show the mean ± SE of the metrics across Fitzpatrick bins 1–2, 3–4, and 5–6 with the number of photographs (in parenthesis). (a) Quality measured by F1 score and precision. (b) Accuracy of severity prediction evaluated with RMSE. EASI, Eczema Area and Severity Index; RMSE, root mean square error; SASSAD, Six Area, Six Sign Atopic Dermatitis; SE, standard error; TISS, Three-Item Severity score.