A protocol for annotation of total body photography for machine learning to analyze skin phenotype and lesion classification

Abstract

Introduction: Artificial Intelligence (AI) has proven effective in classifying skin cancers using dermoscopy images. In experimental settings, algorithms have outperformed expert dermatologists in classifying melanoma and keratinocyte cancers. However, clinical application is limited when algorithms are presented with 'untrained' or out-of-distribution lesion categories, often misclassifying benign lesions as malignant, or misclassifying malignant lesions as benign. Another limitation often raised is the lack of clinical context (e.g., medical history) used as input for the AI decision process. The increasing use of Total Body Photography (TBP) in clinical examinations presents new opportunities for AI to perform holistic analysis of the whole patient, rather than a single lesion. Currently there is a lack of existing literature or standards for image annotation of TBP, or on preserving patient privacy during the machine learning process.

Methods: This protocol describes the methods for the acquisition of patient data, including TBP, medical history, and genetic risk factors, to create a comprehensive dataset for machine learning. 500 patients of various risk profiles will be recruited from two clinical sites (Australia and Spain), to undergo temporal total body imaging, complete surveys on sun behaviors and medical history, and provide a DNA sample. This patient-level metadata is applied to image datasets using DICOM labels. Anonymization and masking methods are applied to preserve patient privacy. A two-step annotation process is followed to label skin images for lesion detection and classification using deep learning models. Skin phenotype characteristics are extracted from images, including innate and facultative skin color, nevi distribution, and UV damage. Several algorithms will be developed relating to skin lesion detection, segmentation and classification, 3D mapping, change detection, and risk profiling. Simultaneously, explainable AI (XAI) methods will be incorporated to foster clinician and patient trust. Additionally, a publicly released dataset of anonymized annotated TBP images will be released for an international challenge to advance the development of new algorithms using this type of data.

Conclusion: The anticipated results from this protocol are validated AI-based tools to provide holistic risk assessment for individual lesions, and risk stratification of patients to assist clinicians in monitoring for skin cancer.

Keywords: artificial intelligence; computer image analyses; computer—aided diagnosis; dermatology; melanoma; total body photography.

Figure 1

Workflow of data management for ML using multimodal data. Recruitment of patients (clinical studies); collection of images (TBP and dermoscopic images) and metadata (medical records and genetics); anonymization; annotation by HITers and dermatologists; dataset for ML training, validation and test; holistic risk score for individuals and individual lesions.

Figure 2

Work view in V7 software, enabling annotation and classification of lesions and sun damage. Symbols on the left side panel represent annotation tools available, allowing users to generate new annotations or edit existing lesion masks. All annotations or masks applied to the image are visible on the right-side pane. The red areas on image tiles are denoted as “area not valid,” as they are better represented for annotation in another image tile. One sun damage (SD) tag can be applied per image based on assessment of the valid area.

Figure 3

Algorithm steps for skin masking (A), lesion detection (B) and lesion segmentation (C). A three-step approach is used for lesion detection and segmentation. (A) as image tiles may include background of non-skin area, a skin mask obtained by thresholding of HSV color space, is applied to minimize inaccurate detections on non-skin area. (B) YOLO (You Only Look Once) (27) model, pretrained on COCO (Common Objects in Contexts) (28) dataset, is adapted to our datasets to detect skin lesions, depicted as bounding boxes. (C) The bounding box predictions are used as prompts for segmentation, leveraging SAM (Segment Anything Model) (29) model to provide precise segmentation of skin lesions.

Figure 4

Steps involved in inter-exploration lesion matching and change detection. (A) First image exploration (i.e., image capture). (B) Second image exploration (e.g., 6–12 months later). (C) Feature matching between sequential explorations. (D) Transformed first image exploration to match second image exploration. (E) Segmentation mask for first image exploration. (F) Segmentation mask for second image exploration. (G) Detected differences between explorations.

Figure 5

Color detection of selected skin lesions. Two examples of original skin lesion images (A,B), and subsequent color detection applied (C,D). This step is used to compare sequential explorations of lesions to detect changes in color.