Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May 20;31(6):1341-1347.
doi: 10.1093/jamia/ocae067.

Can large language models provide secondary reliable opinion on treatment options for dermatological diseases?

Usman Iqbal  1   2   3 , Leon Tsung-Ju Lee  4   5   6 , Annisa Ristya Rahmanti  7   8   9 , Leo Anthony Celi  10   11   12 , Yu-Chuan Jack Li  7   8   13   14
Affiliations

Can large language models provide secondary reliable opinion on treatment options for dermatological diseases?

Usman Iqbal et al. J Am Med Inform Assoc. .

Abstract

Objective: To investigate the consistency and reliability of medication recommendations provided by ChatGPT for common dermatological conditions, highlighting the potential for ChatGPT to offer second opinions in patient treatment while also delineating possible limitations.

Materials and methods: In this mixed-methods study, we used survey questions in April 2023 for drug recommendations generated by ChatGPT with data from secondary databases, that is, Taiwan's National Health Insurance Research Database and an US medical center database, and validated by dermatologists. The methodology included preprocessing queries, executing them multiple times, and evaluating ChatGPT responses against the databases and dermatologists. The ChatGPT-generated responses were analyzed statistically in a disease-drug matrix, considering disease-medication associations (Q-value) and expert evaluation.

Results: ChatGPT achieved a high 98.87% dermatologist approval rate for common dermatological medication recommendations. We evaluated its drug suggestions using the Q-value, showing that human expert validation agreement surpassed Q-value cutoff-based agreement. Varying cutoff values for disease-medication associations, a cutoff of 3 achieved 95.14% accurate prescriptions, 5 yielded 85.42%, and 10 resulted in 72.92%. While ChatGPT offered accurate drug advice, it occasionally included incorrect ATC codes, leading to issues like incorrect drug use and type, nonexistent codes, repeated errors, and incomplete medication codes.

Conclusion: ChatGPT provides medication recommendations as a second opinion in dermatology treatment, but its reliability and comprehensiveness need refinement for greater accuracy. In the future, integrating a medical domain-specific knowledge base for training and ongoing optimization will enhance the precision of ChatGPT's results.

Keywords: ChatGPT; artificial intelligence; decision-making support; dermatology; large language model; medication; second opinion.

PubMed Disclaimer

Conflict of interest statement

None declared.

Figures

Figure 1.
Figure 1.
Workflow for AI-generated disease-drug consultation analysis in dermatology. ATC code, Anatomical Therapeutic Chemical code; ICD-10 codes, International Classification of Diseases-10 codes.
Figure 2.
Figure 2.
Distribution of the top 10 dermatological conditions generated by ChatGPT.
See this image and copyright information in PMC

References

    1. Haug CJ, Drazen JM.. Artificial intelligence and machine learning in clinical medicine, 2023. N Engl J Med. 2023;388(13):1201-1208. 10.1056/NEJMra2302038 - DOI - PubMed
    1. The Lancet Digital Health. ChatGPT: friend or foe? Lancet Digit Health. 2023;5(3):e102. 10.1016/s2589-7500(23)00023-7 - DOI - PubMed
    1. Mello MM, Guha N.. ChatGPT and physicians’ malpractice risk. JAMA Health Forum. 2023;4(5):e231938. 10.1001/jamahealthforum.2023.1938 - DOI - PubMed
    1. Jeblick K, Schachtner B, Dexl J, et al. ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur Radiol. 2023. 10.1007/s00330-023-10213-1 - DOI - PMC - PubMed
    1. Patel SB, Lam K.. ChatGPT: the future of discharge summaries? Lancet Digit Health. 2023;5(3):e107-e108. 10.1016/s2589-7500(23)00021-3 - DOI - PubMed