Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group

doi:10.1001/jamadermatol.2021.4915

. 2022 Jan 1;158(1):90-96.

doi: 10.1001/jamadermatol.2021.4915.

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group

Roxana Daneshjou^{1

2}, Catarina Barata³, Brigid Betz-Stablein⁴, M Emre Celebi⁵, Noel Codella⁶, Marc Combalia⁷, Pascale Guitera^{8

9}, David Gutman¹⁰, Allan Halpern¹¹, Brian Helba¹², Harald Kittler¹³, Kivanc Kose¹¹, Konstantinos Liopyris¹⁴, Josep Malvehy⁷, Han Seung Seog^{15

16}, H Peter Soyer⁴, Eric R Tkaczyk^{17

18

19}, Philipp Tschandl¹³, Veronica Rotemberg¹¹

Affiliations

¹ Stanford Department of Dermatology, Stanford School of Medicine, Redwood City, California.
² Stanford Department of Biomedical Data Science, Stanford School of Medicine, Stanford, California.
³ Institute for Systems and Robotics, Instituto Superior Tecnico, Lisboa, Portugal.
⁴ The University of Queensland Diamantina Institute, The University of Queensland, Dermatology Research Centre, Brisbane, Australia.
⁵ Department of Computer Science and Engineering, University of Central Arkansas, Conway.
⁶ Microsoft, Seattle, Washington.
⁷ Melanoma Unit, Dermatology Department, Hospital Clínic Barcelona, Universitat de Barcelona, IDIBAPS, Barcelona, Spain.
⁸ Melanoma Institute Australia, the University of Sydney, Camperdown, Australia.
⁹ Sydney Melanoma Diagnostic Centre, Royal Prince Alfred Hospital, Camperdown, Australia.
¹⁰ Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
¹¹ Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, New York.
¹² Kitware, Inc, Clifton Park, New York.
¹³ Department of Dermatology, Medical University of Vienna, Vienna, Austria.
¹⁴ University of Athens Medical School, Athens, Greece.
¹⁵ Department of Dermatology, I Dermatology Clinic, Seoul, Korea.
¹⁶ IDerma, Inc, Seoul, Korea.
¹⁷ Dermatology Service and Research Service, Tennessee Valley Healthcare System, Department of Veterans Affairs, Nashville.
¹⁸ Vanderbilt Dermatology Translational Research Clinic, Department of Dermatology, Vanderbilt University Medical Center, Nashville, Tennessee.
¹⁹ Department of Biomedical Engineering, Vanderbilt University, Nashville, Tennessee.

PMID: 34851366
PMCID: PMC9845064
DOI: 10.1001/jamadermatol.2021.4915

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group

Roxana Daneshjou et al. JAMA Dermatol. 2022.

. 2022 Jan 1;158(1):90-96.

doi: 10.1001/jamadermatol.2021.4915.

Authors

Affiliations

¹ Stanford Department of Dermatology, Stanford School of Medicine, Redwood City, California.
² Stanford Department of Biomedical Data Science, Stanford School of Medicine, Stanford, California.
³ Institute for Systems and Robotics, Instituto Superior Tecnico, Lisboa, Portugal.
⁴ The University of Queensland Diamantina Institute, The University of Queensland, Dermatology Research Centre, Brisbane, Australia.
⁵ Department of Computer Science and Engineering, University of Central Arkansas, Conway.
⁶ Microsoft, Seattle, Washington.
⁷ Melanoma Unit, Dermatology Department, Hospital Clínic Barcelona, Universitat de Barcelona, IDIBAPS, Barcelona, Spain.
⁸ Melanoma Institute Australia, the University of Sydney, Camperdown, Australia.
⁹ Sydney Melanoma Diagnostic Centre, Royal Prince Alfred Hospital, Camperdown, Australia.
¹⁰ Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
¹¹ Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, New York.
¹² Kitware, Inc, Clifton Park, New York.
¹³ Department of Dermatology, Medical University of Vienna, Vienna, Austria.
¹⁴ University of Athens Medical School, Athens, Greece.
¹⁵ Department of Dermatology, I Dermatology Clinic, Seoul, Korea.
¹⁶ IDerma, Inc, Seoul, Korea.
¹⁷ Dermatology Service and Research Service, Tennessee Valley Healthcare System, Department of Veterans Affairs, Nashville.
¹⁸ Vanderbilt Dermatology Translational Research Clinic, Department of Dermatology, Vanderbilt University Medical Center, Nashville, Tennessee.
¹⁹ Department of Biomedical Engineering, Vanderbilt University, Nashville, Tennessee.

PMID: 34851366
PMCID: PMC9845064
DOI: 10.1001/jamadermatol.2021.4915

Abstract

Importance: The use of artificial intelligence (AI) is accelerating in all aspects of medicine and has the potential to transform clinical care and dermatology workflows. However, to develop image-based algorithms for dermatology applications, comprehensive criteria establishing development and performance evaluation standards are required to ensure product fairness, reliability, and safety.

Objective: To consolidate limited existing literature with expert opinion to guide developers and reviewers of dermatology AI.

Evidence review: In this consensus statement, the 19 members of the International Skin Imaging Collaboration AI working group volunteered to provide a consensus statement. A systematic PubMed search was performed of English-language articles published between December 1, 2008, and August 24, 2021, for "artificial intelligence" and "reporting guidelines," as well as other pertinent studies identified by the expert panel. Factors that were viewed as critical to AI development and performance evaluation were included and underwent 2 rounds of electronic discussion to achieve consensus.

Findings: A checklist of items was developed that outlines best practices of image-based AI development and assessment in dermatology.

Conclusions and relevance: Clinically effective AI needs to be fair, reliable, and safe; this checklist of best practices will help both developers and reviewers achieve this goal.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest:

Roxana Daneshjou has done consulting for VisualDx, DWA, Pfizer and has research funded by UCB. H. Peter Soyer is a shareholder of MoleMap NZ Limited and e-derm consult GmbH, and undertakes regular teledermatological reporting for both companies. H. Peter Soyer is a Medical Consultant for Canfield Scientific Inc., MoleMap Australia Pty Limited, a Medical Advisor for First Derm and Revenio Research Oy. Han Seung Seog is the founder of IDerma (https://ai.skindx.net). Philipp Tschandl reports grants from Lilly and MetaOptima, outside the submitted work, and consultation and/or speaker fees from Lilly, Silverchair, FotoFinder and Novartis, outside the submitted work. Pascale Guitera received some honoraria form Metaoptima PTY. Noel Codella is an employee of Microsoft. Brian Helba is an employee of Kitware, Inc.

Comment in

Image Consent and the Development of Image-Based Artificial Intelligence-Reply.
Daneshjou R, Rotemberg V; International Skin Imaging Collaboration Artificial Intelligence Working Group. Daneshjou R, et al. JAMA Dermatol. 2022 May 1;158(5):590. doi: 10.1001/jamadermatol.2022.0108. JAMA Dermatol. 2022. PMID: 35416913 No abstract available.
Image Consent and the Development of Image-Based Artificial Intelligence.
Kovarik CL, Sanabria B, Stoff BK. Kovarik CL, et al. JAMA Dermatol. 2022 May 1;158(5):589. doi: 10.1001/jamadermatol.2022.0689. JAMA Dermatol. 2022. PMID: 35416918 No abstract available.

Cited by

A dataset of skin lesion images collected in Argentina for the evaluation of AI tools in this population.
Ricci Lara MA, Rodríguez Kowalczuk MV, Lisa Eliceche M, Ferraresso MG, Luna DR, Benitez SE, Mazzuoccolo LD. Ricci Lara MA, et al. Sci Data. 2023 Oct 18;10(1):712. doi: 10.1038/s41597-023-02630-0. Sci Data. 2023. PMID: 37853053 Free PMC article.
Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis.
Krakowski I, Kim J, Cai ZR, Daneshjou R, Lapins J, Eriksson H, Lykou A, Linos E. Krakowski I, et al. NPJ Digit Med. 2024 Apr 9;7(1):78. doi: 10.1038/s41746-024-01031-w. NPJ Digit Med. 2024. PMID: 38594408 Free PMC article. Review.
Recent Advances in Melanoma Diagnosis and Prognosis Using Machine Learning Methods.
Grossarth S, Mosley D, Madden C, Ike J, Smith I, Huo Y, Wheless L. Grossarth S, et al. Curr Oncol Rep. 2023 Jun;25(6):635-645. doi: 10.1007/s11912-023-01407-3. Epub 2023 Mar 31. Curr Oncol Rep. 2023. PMID: 37000340 Free PMC article. Review.
Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review.
Wang Y, Li N, Chen L, Wu M, Meng S, Dai Z, Zhang Y, Clarke M. Wang Y, et al. J Med Internet Res. 2023 Nov 22;25:e46089. doi: 10.2196/46089. J Med Internet Res. 2023. PMID: 37991819 Free PMC article.
Artificial Intelligence in Diagnostic Dermatology: Challenges and the Way Forward.
Sengupta D. Sengupta D. Indian Dermatol Online J. 2023 Oct 17;14(6):782-787. doi: 10.4103/idoj.idoj_462_23. eCollection 2023 Nov-Dec. Indian Dermatol Online J. 2023. PMID: 38099026 Free PMC article. Review.

See all "Cited by" articles

References

1. Daneshjou R, He B, Ouyang D, Zou JY. How to evaluate deep learning for cancer diagnostics - factors and recommendations. Biochim Biophys Acta Rev Cancer. Apr 2021;1875(2):188515. doi:10.1016/j.bbcan.2021.188515 - DOI - PMC - PubMed
1. Wawira Gichoya J, McCoy LG, Celi LA, Ghassemi M. Equity in essence: a call for operationalising fairness in machine learning for healthcare. BMJ Health Care Inform. Apr 2021;28(1)doi:10.1136/bmjhci-2020-100289 - DOI - PMC - PubMed
1. Taylor M, Liu X, Denniston A, et al. Raising the Bar for Randomized Trials Involving Artificial Intelligence: The SPIRIT-Artificial Intelligence and CONSORT-Artificial Intelligence Guidelines. J Invest Dermatol. Mar 2021;doi:10.1016/j.jid.2021.02.744 - DOI - PubMed
1. Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, Group S-AaC- AW. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. Sep 2020;26(9):1364–1374. doi:10.1038/s41591-020-1034-x - DOI - PMC - PubMed
1. Group D- AS. DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med. 02 2021;27(2):186–187. doi:10.1038/s41591-021-01229-5 - DOI - PubMed

Publication types

Actions
Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources

[1] Daneshjou R, He B, Ouyang D, Zou JY. How to evaluate deep learning for cancer diagnostics - factors and recommendations. Biochim Biophys Acta Rev Cancer. Apr 2021;1875(2):188515. doi:10.1016/j.bbcan.2021.188515 - DOI - PMC - PubMed

[2] Daneshjou R, He B, Ouyang D, Zou JY. How to evaluate deep learning for cancer diagnostics - factors and recommendations. Biochim Biophys Acta Rev Cancer. Apr 2021;1875(2):188515. doi:10.1016/j.bbcan.2021.188515 - DOI - PMC - PubMed

[3] Wawira Gichoya J, McCoy LG, Celi LA, Ghassemi M. Equity in essence: a call for operationalising fairness in machine learning for healthcare. BMJ Health Care Inform. Apr 2021;28(1)doi:10.1136/bmjhci-2020-100289 - DOI - PMC - PubMed

[4] Wawira Gichoya J, McCoy LG, Celi LA, Ghassemi M. Equity in essence: a call for operationalising fairness in machine learning for healthcare. BMJ Health Care Inform. Apr 2021;28(1)doi:10.1136/bmjhci-2020-100289 - DOI - PMC - PubMed

[5] Taylor M, Liu X, Denniston A, et al. Raising the Bar for Randomized Trials Involving Artificial Intelligence: The SPIRIT-Artificial Intelligence and CONSORT-Artificial Intelligence Guidelines. J Invest Dermatol. Mar 2021;doi:10.1016/j.jid.2021.02.744 - DOI - PubMed

[6] Taylor M, Liu X, Denniston A, et al. Raising the Bar for Randomized Trials Involving Artificial Intelligence: The SPIRIT-Artificial Intelligence and CONSORT-Artificial Intelligence Guidelines. J Invest Dermatol. Mar 2021;doi:10.1016/j.jid.2021.02.744 - DOI - PubMed

[7] Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, Group S-AaC- AW. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. Sep 2020;26(9):1364–1374. doi:10.1038/s41591-020-1034-x - DOI - PMC - PubMed

[8] Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, Group S-AaC- AW. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med. Sep 2020;26(9):1364–1374. doi:10.1038/s41591-020-1034-x - DOI - PMC - PubMed

[9] Group D- AS. DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med. 02 2021;27(2):186–187. doi:10.1038/s41591-021-01229-5 - DOI - PubMed

[10] Group D- AS. DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med. 02 2021;27(2):186–187. doi:10.1038/s41591-021-01229-5 - DOI - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group

Affiliations

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group

Authors

Affiliations

Abstract

Conflict of interest statement

Comment in

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Abstract

Conflict of interest statement

Comment in

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources