Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Dec;90(12):3361-3366.
doi: 10.1111/bcp.16275. Epub 2024 Oct 2.

Evaluating the capability of ChatGPT in predicting drug-drug interactions: Real-world evidence using hospitalized patient data

Ramya Padmavathy Radha Krishnan  1 , Euniss Hinyo Hung  1   2 , Megan Ashford  2 , Clark Ethan Edillo  2 , Charlise Gardner  2 , Hector Blake Hatrick  2 , Byungjun Kim  1   2 , Angel Wing Yan Lai  1 , Xinran Li  1   2 , Yvonne Xinyi Zhao  1   2 , Jacques Eugene Raubenheimer  1
Affiliations

Evaluating the capability of ChatGPT in predicting drug-drug interactions: Real-world evidence using hospitalized patient data

Ramya Padmavathy Radha Krishnan et al. Br J Clin Pharmacol. 2024 Dec.

Abstract

Drug-drug interactions (DDIs) present a significant health burden, compounded by clinician time constraints and poor patient health literacy. We assessed the ability of ChatGPT (generative artificial intelligence-based large language model) to predict DDIs in a real-world setting. Demographics, diagnoses and prescribed medicines for 120 hospitalized patients were input through three standardized prompts to ChatGPT version 3.5 and compared against pharmacist DDI evaluation to estimate diagnostic accuracy. Area under receiver operating characteristic and inter-rater reliability (Cohen's and Fleiss' kappa coefficients) were calculated. ChatGPT's responses differed based on prompt wording style, with higher sensitivity for prompts mentioning 'drug interaction'. Confusion matrices displayed low true positive and high true negative rates, and there was minimal agreement between ChatGPT and pharmacists (Cohen's kappa values 0.077-0.143). Low sensitivity values suggest a lack of success in identifying DDIs by ChatGPT, and further development is required before it can reliably assess potential DDIs in real-world scenarios.

Keywords: ChatGPT; artificial intelligence (AI); drug–drug interaction (DDI); large language model (LLM); patient health literacy.

PubMed Disclaimer

Conflict of interest statement

The authors report no conflict of interest.

Figures

FIGURE 1
FIGURE 1
Set of confusion matrices for each prompt and all three combined. The top left and right quadrants are true negative and false positive values, respectively. Bottom left and right quadrants are false negative and true positive values, respectively. P1: prompt 1, P2: prompt 2, P3: prompt 3.
FIGURE 2
FIGURE 2
Sensitivity and specificity values for each prompt and all three combined. P1: prompt 1, P2: prompt 2, P3: prompt 3.
See this image and copyright information in PMC

References

    1. Snyder BD, Polasek TM, Doogue MP. Drug interactions: principles and practice. Aust Prescr. 2012;35(3):85‐88. doi:10.18773/austprescr.2012.037 - DOI
    1. Ajayi FO, Sun H, Perry J. Adverse drug reactions: a review of relevant factors. J Clin Pharmacol. 2000;40(10):1093‐1101. doi:10.1177/009127000004001003 - DOI - PubMed
    1. Delara M, Murray L, Jafari B, et al. Prevalence and factors associated with polypharmacy: a systematic review and meta‐analysis. BMC Geriatr. 2022;22(1):601. doi:10.1186/s12877-022-03279-x - DOI - PMC - PubMed
    1. Kim J, Parish AL. Polypharmacy and medication management in older adults. Nurs Clin North Am. 2017;52(3):457‐468. doi:10.1016/j.cnur.2017.04.007 - DOI - PubMed
    1. Pazan F, Wehling M. Polypharmacy in older adults: a narrative review of definitions, epidemiology and consequences. Eur Geriatr Med. 2021;12(3):443‐452. doi:10.1007/s41999-021-00479-3 - DOI - PMC - PubMed