Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May 20;31(6):1388-1396.
doi: 10.1093/jamia/ocae041.

Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support

Siru Liu  1   2 , Allison B McCoy  1 , Aileen P Wright  1   3 , Scott D Nelson  1 , Sean S Huang  1   3 , Hasan B Ahmad  4 , Sabrina E Carro  5 , Jacob Franklin  3 , James Brogan  3 , Adam Wright  1   3
Affiliations

Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support

Siru Liu et al. J Am Med Inform Assoc. .

Abstract

Objectives: To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts.

Materials and methods: We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness.

Results: Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001).

Conclusion: End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.

Keywords: alert fatigue; clinical decision support; health personnel; large language model.

PubMed Disclaimer

Conflict of interest statement

S.N. is on the advisory board for Baxter Health and Merative Micromedex. The other authors do not have conflicts of interest related to this study.

Figures

Figure 1.
Figure 1.
An example of a CDS alert and relevant user comments. CDS = clinical decision support.
Figure 2.
Figure 2.
Stacked bar charts of the scores for clarity, completeness, accuracy, and usefulness of AI-generated summaries (A) and human-generated summaries (B). AI = artificial intelligence.
Figure 3.
Figure 3.
Using AI-generated summary of user comments to improve CDS alerts offline and online. AI = artificial intelligence, CDS = clinical decision support.
See this image and copyright information in PMC

References

    1. Parasrampuria S, Henry J.. Hospitals’ use of electronic health records data, 2015-2017. ONC Data Br. 2019;46:1-13. - PubMed
    1. Wright A, Sittig DF, Ash JS, et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Informatics Assoc. 2011;18(3):232-242. 10.1136/amiajnl-2011-000113 - DOI - PMC - PubMed
    1. Thomas Craig KJ, Fusco N, Lindsley K, et al. Rapid review: identification of digital health interventions in atherosclerotic-related cardiovascular disease populations to address racial, ethnic, and socioeconomic health disparities. Cardiovasc Digit Health J. 2020;1(3):139-148. 10.1016/j.cvdhj.2020.11.001 - DOI - PMC - PubMed
    1. Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331-345. 10.4338/ACI-2010-05-RA-0031 - DOI - PMC - PubMed
    1. Douthit BJ, McCoy AB, Nelson SD.. The impact of clinical decision support on health disparities and the digital divide. Yearb Med Inform. 2023;32(1):169-178. Published Online First: 6 July 10.1055/s-0043-1768722 - DOI - PMC - PubMed