Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jul 2;23(7):e20708.
doi: 10.2196/20708.

Integrating Patient Data Into Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review

Julia Höhn #  1 , Achim Hekler #  1 , Eva Krieghoff-Henning  1 , Jakob Nikolas Kather  2   3 , Jochen Sven Utikal  4   5 , Friedegund Meier  6 , Frank Friedrich Gellrich  6 , Axel Hauschild  7 , Lars French  8 , Justin Gabriel Schlager  8 , Kamran Ghoreschi  9 , Tabea Wilhelm  9 , Heinz Kutzner  10 , Markus Heppt  11 , Sebastian Haferkamp  12 , Wiebke Sondermann  13 , Dirk Schadendorf  13 , Bastian Schilling  14 , Roman C Maron  1 , Max Schmitt  1 , Tanja Jutzi  1 , Stefan Fröhling  3   15 , Daniel B Lipka  3   15   16 , Titus Josef Brinker  1
Affiliations

Integrating Patient Data Into Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review

Julia Höhn et al. J Med Internet Res. .

Abstract

Background: Recent years have been witnessing a substantial improvement in the accuracy of skin cancer classification using convolutional neural networks (CNNs). CNNs perform on par with or better than dermatologists with respect to the classification tasks of single images. However, in clinical practice, dermatologists also use other patient data beyond the visual aspects present in a digitized image, further increasing their diagnostic accuracy. Several pilot studies have recently investigated the effects of integrating different subtypes of patient data into CNN-based skin cancer classifiers.

Objective: This systematic review focuses on the current research investigating the impact of merging information from image features and patient data on the performance of CNN-based skin cancer image classification. This study aims to explore the potential in this field of research by evaluating the types of patient data used, the ways in which the nonimage data are encoded and merged with the image features, and the impact of the integration on the classifier performance.

Methods: Google Scholar, PubMed, MEDLINE, and ScienceDirect were screened for peer-reviewed studies published in English that dealt with the integration of patient data within a CNN-based skin cancer classification. The search terms skin cancer classification, convolutional neural network(s), deep learning, lesions, melanoma, metadata, clinical information, and patient data were combined.

Results: A total of 11 publications fulfilled the inclusion criteria. All of them reported an overall improvement in different skin lesion classification tasks with patient data integration. The most commonly used patient data were age, sex, and lesion location. The patient data were mostly one-hot encoded. There were differences in the complexity that the encoded patient data were processed with regarding deep learning methods before and after fusing them with the image features for a combined classifier.

Conclusions: This study indicates the potential benefits of integrating patient data into CNN-based diagnostic algorithms. However, how exactly the individual patient data enhance classification performance, especially in the case of multiclass classification problems, is still unclear. Moreover, a substantial fraction of patient data used by dermatologists remains to be analyzed in the context of CNN-based skin cancer classification. Further exploratory analyses in this promising field may optimize patient data integration into CNN-based skin cancer diagnostics for patients' benefits.

Keywords: convolutional neural networks; patient data; skin cancer classification.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: A Hauschild reports clinical trial support, speaker’s honoraria, and consultancy fees from the following companies: Amgen, Bristol Myers Squibb (BMS), Merck Serono, Merck Sharp & Dohme (MSD), Novartis, Oncosec, Philogen, Pierre Fabre, Provectus, Regeneron, Roche, OncoSec, Sanofi Genzyme, and Sun Pharma (outside the submitted work). BS reports advisory roles for or has received honoraria from Pierre Fabre Pharmaceuticals, Incyte, Novartis, Roche, BMS, and MSD; research funding from BMS, Pierre Fabre Pharmaceuticals and MSD; and travel support from Novartis, Roche, BMS, Pierre Fabre Pharmaceuticals, and Amgen outside the submitted work. FM has received travel support, speaker’s fees, and/or advisor’s honoraria from Novartis, Roche, BMS, MSD, and Pierre Fabre and research funding from Novartis and Roche outside the submitted work. JSU is on the advisory board or has received honoraria and travel support from Amgen, Bristol Myers Squibb, GSK, LeoPharma, Merck Sharp and Dohme, Novartis, Pierre Fabre, Roche, outside the submitted work. SH reports advisory roles for or has received honoraria from Pierre Fabre Pharmaceuticals, Novartis, Roche, BMS, Amgen, and MSD outside the submitted work. TJB reports owning a company that develops mobile apps (Smart Health Heidelberg GmbH, Handschuhsheimer Landstr. 9/1, 69120 Heidelberg). WS received travel expenses for attending meetings and/or (speaker) honoraria from Abbvie, Almirall, Bristol Myers Squibb, Celgene, Janssen, LEO Pharma, Lilly, MSD, Novartis, Pfizer, Roche, Sanofi Genzyme, and UCB outside the submitted work.

Figures

Figure 1
Figure 1
An overview of patient data considered by dermatologists while diagnosing skin lesions. The framed characteristics in the figure illustrate the fraction of patient data that can potentially be recognized by convolutional neural networks from a single image input. UVR: ultraviolet radiation.
Figure 2
Figure 2
Overview of the different fusing techniques in the main function blocks of the combined classifier. CNN: convolutional neural network.
See this image and copyright information in PMC

References

    1. Schadendorf D, van Akkooi AC, Berking C, Griewank KG, Gutzmer R, Hauschild A, Stang A, Roesch A, Ugurel S. Melanoma. Lancet. 2018 Sep;392(10151):971–84. doi: 10.1016/s0140-6736(18)31559-9. - DOI - PubMed
    1. Brinker TJ, Hekler A, Hauschild A, Berking C, Schilling B, Enk AH, Haferkamp S, Karoglan A, von Kalle C, Weichenthal M, Sattler E, Schadendorf D, Gaiser MR, Klode J, Utikal JS. Comparing artificial intelligence algorithms to 157 German dermatologists: the melanoma classification benchmark. Eur J Cancer. 2019 Apr;111:30–7. doi: 10.1016/j.ejca.2018.12.016. https://linkinghub.elsevier.com/retrieve/pii/S0959-8049(18)31562-4 - DOI - PubMed
    1. Dick V, Sinz C, Mittlböck M, Kittler H, Tschandl P. Accuracy of computer-aided diagnosis of melanoma: a meta-analysis. JAMA Dermatol. 2019 Nov 01;155(11):1291–9. doi: 10.1001/jamadermatol.2019.1375. http://europepmc.org/abstract/MED/31215969 - DOI - PMC - PubMed
    1. Brinker TJ, Hekler A, Utikal JS, Grabe N, Schadendorf D, Klode J, Berking C, Steeb T, Enk AH, von Kalle C. Skin cancer classification using convolutional neural networks: systematic review. J Med Internet Res. 2018 Oct 17;20(10):e11936. doi: 10.2196/11936. https://www.jmir.org/2018/10/e11936/ - DOI - PMC - PubMed
    1. Rawat W, Wang Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 2017 Sep;29(9):2352–449. doi: 10.1162/neco_a_00990. - DOI - PubMed

Publication types