Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May 15:10:20552076241253523.
doi: 10.1177/20552076241253523. eCollection 2024 Jan-Dec.

Safety and quality of AI chatbots for drug-related inquiries: A real-world comparison with licensed pharmacists

Yasser Albogami  1 , Almaha Alfakhri  2 , Abdulaziz Alaqil  3 , Aljawharah Alkoraishi  4 , Heba Alshammari  4 , Yasmin Elsharawy  4 , Abdullah Alhammad  1 , Abdulaziz Alhossan  1
Affiliations

Safety and quality of AI chatbots for drug-related inquiries: A real-world comparison with licensed pharmacists

Yasser Albogami et al. Digit Health. .

Abstract

Introduction: Pharmacists play a pivotal role in ensuring patients are administered safe and effective medications; however, they encounter obstacles such as elevated workloads and a scarcity of qualified professionals. Despite the prospective utility of large language models (LLMs), such as Generative Pre-trained Transformers (GPTs), in addressing pharmaceutical inquiries, their applicability in real-world cases remains unexplored.

Objective: To evaluate GPT-based chatbots' accuracy in real-world drug-related inquiries, comparing their performance to licensed pharmacists.

Methods: In this cross-sectional study, authors analyzed real-world drug inquiries from a Drug Information Inquiry Database. Two independent pharmacists evaluated the performance of GPT-based chatbots (GPT-3, GPT-3.5, GPT-4) against human pharmacists using accuracy, detail, and risk of harm criteria. Descriptive statistics described inquiry characteristics. Absolute proportion comparative analyses assessed accuracy, detail, and risk of harm. Stratified analyses were performed for different inquiry types.

Results: Seventy inquiries were included. Most inquiries were received from physicians (41%) and pharmacists (44%). Inquiries type included dosage/administration (34.2%), drug interaction (12.8%) and pregnancy/lactation (15.7%). Majority of inquires included adults (83%) and female patients (54.3%). GPT-4 had 64.3% completely accurate responses, comparable to human pharmacists. GPT-4 and human pharmacists provided sufficiently detailed responses, with GPT-4 offering additional relevant details. Both GPT-4 and human pharmacists delivered 95% safe responses; however, GPT-4 provided proactive risk mitigation information in 70% of the instances, whereas similar information was included in 25.7% of human pharmacists' responses.

Conclusion: Our study showcased GPT-4's potential in addressing drug-related inquiries accurately and safely, comparable to human pharmacists. Current GPT-4-based chatbots could support healthcare professionals and foster global health improvements.

Keywords: GPT-4; Pharmacists; artificial intelligence; real-world data.

PubMed Disclaimer

Conflict of interest statement

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The views expressed in this paper are those of the author(s) and not do not necessarily reflect those of the SFDA or its stakeholders. Guaranteeing the accuracy and the validity of the data is a sole responsibility of the research team.

Figures

Figure 1.
Figure 1.
Accuracy of GPT models and human pharmacists’ responses to drug-related inquiries.
Figure 2.
Figure 2.
Details of GPT models and human pharmacists’ responses to drug-related inquiries.
Figure 3.
Figure 3.
Risk of harm in GPT models and human pharmacists’ responses to drug-related inquiries.
See this image and copyright information in PMC

References

    1. Ghaibi S, Ipema H, Gabay M. ASHP Guidelines on the pharmacist’s role in providing drug information. Am J Health-Syst Pharm 2015; 72: 573–577. - PubMed
    1. Dee J, Dhuhaibawi N, Hayden JC. A systematic review and pooled prevalence of burnout in pharmacists. Int J Clin Pharm 2023; 45: 1027–1036. - PMC - PubMed
    1. Meštrović A, Al-Haqan A, El-Akel M, et al. Connecting the dots in pharmacy education: the FIP international pharmaceutical federation global competency framework for educators and trainers in pharmacy (FIP-GCFE). Pharmacy Education 2022; 22: 100–109.
    1. Bates I, John C, Seegobin Pet al. et al. An analysis of the global pharmacy workforce capacity trends from 2006 to 2012. Hum Resour Health 2018; 16: 3. - PMC - PubMed
    1. Dale R. GPT-3: what’s it good for? Nat Lang Eng 2021; 27: 113–118.

LinkOut - more resources