Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Aug;80(8):1133-1140.
doi: 10.1007/s00228-024-03687-5. Epub 2024 Apr 9.

Poor performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction

Merel van Nuland  1 , JaapJan D Snoep  2 , Toine Egberts  3   4 , Abdullah Erdogan  5 , Ricky Wassink  5 , Paul D van der Linden  5
Affiliations

Poor performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction

Merel van Nuland et al. Eur J Clin Pharmacol. 2024 Aug.

Abstract

Purpose: Clinical decision support systems (CDSS) are used to identify drugs with potential need for dose modification in patients with renal impairment. ChatGPT holds the potential to be integrated in the electronic health record (EHR) system to give such dosing advices. In this study, we aim to evaluate the performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal impairment.

Methods: This cross-sectional study was performed at Tergooi Medical Center, the Netherlands. CDSS alerts regarding renal dysfunction were collected from the electronic health record (EHR) during a 2-week period and were presented to ChatGPT and an expert panel. Alerts were presented with and without patient variables. To evaluate the performance, suggested medication interventions were compared.

Results: In total, 172 CDDS alerts were generated for 80 patients. Indecisive responses by ChatGPT to alerts were excluded. For alerts presented without patient variables, ChatGPT provided "correct and identical" responses to 19.9%, "correct and different" responses to 26.7%, and "incorrect responses to 53.4% of the alerts. For alerts including patient variables, ChatGPT provided "correct and identical" responses to 16.7%, "correct and different" responses to 16.0%, and "incorrect responses to 67.3% of the alerts. Accuracy was better for newer drugs such as direct oral anticoagulants.

Conclusion: The performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction was poor. Based on these results, we conclude that ChatGPT, in its current state, is not appropriate for automatic integration into our EHR to handle CDSS alerts related to renal dysfunction.

Keywords: CDSS; ChatGPT; Clinical rule-guided dose interventions; Language model; Renal dysfunction.

PubMed Disclaimer

Similar articles

See all similar articles

Cited by

References

    1. Plana D, Shung DL, Grimshaw AA et al (2022) Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw Open 5:e2233946. https://doi.org/10.1001/jamanetworkopen.2022.33946 - DOI - PubMed - PMC
    1. OpenIA ChatGPT. https://openai.com/blog/chatgpt . Accessed 26 Jan 2024
    1. Roosan D, Padua P, Khan R et al (2003) Effectiveness of ChatGPT in clinical pharmacy and the role of artificial intelligence in medication therapy management. J Am Pharm Assoc. https://doi.org/10.1016/j.japh.2023.11.023 - DOI
    1. Al-Dujaili Z, Omari S, Pillai J, Al Faraj A (2023) Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: a preliminary analysis with clinical pharmacy experts worldwide. Res Social Adm Pharm 19:1590–1594. https://doi.org/10.1016/j.sapharm.2023.08.012 - DOI - PubMed
    1. Morath B, Chiriac U, Jaszkowski E et al (2023) Performance and risks of ChatGPT used in drug information: an exploratory real-world analysis. Eur J Hosp Pharm. https://doi.org/10.1136/ejhpharm-2023-003750 - DOI - PubMed

LinkOut - more resources