Expert Agreement on the Presence and Spatial Localization of Melanocytic Features in Dermoscopy

Abstract

Dermoscopy aids in melanoma detection; however, agreement on dermoscopic features, including those of high clinical relevance, remains poor. In this study, we attempted to evaluate agreement among experts on exemplar images not only for the presence of melanocytic-specific features but also for spatial localization. This was a cross-sectional, multicenter, observational study. Dermoscopy images exhibiting at least 1 of 31 melanocytic-specific features were submitted by 25 world experts as exemplars. Using a web-based platform that allows for image markup of specific contrast-defined regions (superpixels), 20 expert readers annotated 248 dermoscopic images in collections of 62 images. Each collection was reviewed by five independent readers. A total of 4,507 feature observations were performed. Good-to-excellent agreement was found for 14 of 31 features (45.2%), with eight achieving excellent agreement (Gwet's AC >0.75) and seven of them being melanoma-specific features. These features were peppering/granularity (0.91), shiny white streaks (0.89), typical pigment network (0.83), blotch irregular (0.82), negative network (0.81), irregular globules (0.78), dotted vessels (0.77), and blue-whitish veil (0.76). By utilizing an exemplar dataset, a good-to-excellent agreement was found for 14 features that have previously been shown useful in discriminating nevi from melanoma. All images are public (www.isic-archive.com) and can be used for education, scientific communication, and machine learning experiments.

Figure 1.. Confusability matrix.

Each element of this confusion matrix is the median Dice coefficient across all pairs of features that occurred in the dataset. In the cases where the diagonal elements are as close to 1.0 as possible (dark color), this suggests that all pairs of readers (each individual reader directly compared with each of the other four, across all studies) selected close to identical sets of superpixels for the same feature and image. When all off-diagonal elements approach 0.0 (light color), this suggests that for different features, pairs of readers rarely selected overlapping superpixels. Feature pattern:homogeneous, a nonspecific feature that can be observed in any region of the lesion overlaps with a multitude of features, commonly with blotch:regular and vessels:comma; however, the agreement on the superpixel level for this feature among experts for this feature was 0%. In addition, features dots:regular and dots:irregular overlap among them and with globules/clods, whereas both the superpixel level agreement and Gwet’s AC were low for these features.

Figure 2.. Superpixel annotation of exemplar image.

(a) Sample image (ISIC_0022328) of a melanoma included in the study for exemplar feature structureless:blue—whitish veil. (b–f) Image overlaid with the superpixel outlines and annotation markups selected by each of the five expert readers. (g) Color representation of each feature selected by the readers. (h) Agreement overlaps among all readers.