Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2021 May:149:94-101.
doi: 10.1016/j.ejca.2021.02.032. Epub 2021 Apr 7.

Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification

Julia Höhn  1 , Eva Krieghoff-Henning  1 , Tanja B Jutzi  1 , Christof von Kalle  2 , Jochen S Utikal  3 , Friedegund Meier  4 , Frank F Gellrich  4 , Sarah Hobelsberger  4 , Axel Hauschild  5 , Justin G Schlager  6 , Lars French  6 , Lucie Heinzerling  6 , Max Schlaak  7 , Kamran Ghoreschi  7 , Franz J Hilke  7 , Gabriela Poch  7 , Heinz Kutzner  8 , Markus V Heppt  9 , Sebastian Haferkamp  10 , Wiebke Sondermann  11 , Dirk Schadendorf  11 , Bastian Schilling  12 , Matthias Goebeler  12 , Achim Hekler  1 , Stefan Fröhling  13 , Daniel B Lipka  13 , Jakob N Kather  14 , Dieter Krahl  15 , Gerardo Ferrara  16 , Sarah Haggenmüller  1 , Titus J Brinker  17
Affiliations
Free article
Comparative Study

Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification

Julia Höhn et al. Eur J Cancer. 2021 May.
Free article

Abstract

Background: Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses.

Objectives: We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone.

Methods: We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier.

Results: The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%.

Conclusion: In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy.

Keywords: Convolutional neural networks; Data fusion; Histologic whole slide images; Patient data; Skin cancer classification.

PubMed Disclaimer

Conflict of interest statement

Conflict of interest statement The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Haferkamp S. reports advisory roles for or has received honoraria from Pierre Fabre Pharmaceuticals, Novartis, Roche, BMS, Amgen and MSD outside the submitted work. Hauschild A. reports clinical trial support, speaker's honoraria or consultancy fees from the following companies: Amgen, BMS, Merck Serono, MSD, Novartis, Oncosec, Philogen, Pierre Fabre, Provectus, Regeneron, Roche, OncoSec, Sanofi-Genzyme and Sun Pharma, outside the submitted work. BS reports advisory roles for or has received honoraria from Pierre Fabre Pharmaceuticals, Incyte, Novartis, Roche, BMS and MSD, research funding from BMS, Pierre Fabre Pharmaceuticals and MSD, and travel support from Novartis, Roche, BMS, Pierre Fabre Pharmaceuticals and Amgen, outside the submitted work. JSU is on the advisory board or has received honoraria and travel support from Amgen, Bristol Myers Squibb, GSK, LeoPharma, Merck Sharp and Dohme, Novartis, Pierre Fabre, Roche, outside the submitted work. WS received travel expenses for attending meetings and/or (speaker) honoraria from Abbvie, Almirall, Bristol-Myers Squibb, Celgene, Janssen, LEO Pharma, Lilly, MSD, Novartis, Pfizer, Roche, Sanofi Genzyme and UCB outside the submitted work. FM has received travel support or/and speaker's fees or/and advisor's honoraria by Novartis, Roche, BMS, MSD and Pierre Fabre and research funding from Novartis and Roche. TJB reports owning a company that develops mobile applications (Smart Health Heidelberg GmbH, Handschuhsheimer Landstr. 9/1, 69120 Heidelberg, https://smarthealth.de). The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

MeSH terms

LinkOut - more resources