Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Jun 20;30(7):1237-1245.
doi: 10.1093/jamia/ocad072.

Using AI-generated suggestions from ChatGPT to optimize clinical decision support

Siru Liu  1 , Aileen P Wright  1   2 , Barron L Patterson  3 , Jonathan P Wanderer  1   4 , Robert W Turer  5   6 , Scott D Nelson  1 , Allison B McCoy  1 , Dean F Sittig  7 , Adam Wright  1
Affiliations

Using AI-generated suggestions from ChatGPT to optimize clinical decision support

Siru Liu et al. J Am Med Inform Assoc. .

Abstract

Objective: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions.

Methods: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy.

Results: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy.

Conclusion: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Keywords: artificial intelligence; clinical decision support; large language model.

PubMed Disclaimer

Conflict of interest statement

The authors do not have conflicts of interest related to this study.

Figures

Figure 1.
Figure 1.
An example of using ChatGPT to generate suggestions to improve an alert.
Figure 2.
Figure 2.
Stacked bar charts of the scores of each item (understanding, relevance, usefulness, acceptance, workflow, bias, inversion, redundancy) for AI-generated suggestions (A) and human-generated suggestions (B).
Figure 3.
Figure 3.
A prototype of potential implementation in EHR system—AI decision support editors.
See this image and copyright information in PMC

Comment in

References

    1. Kawamoto K, Houlihan CA, Balas EA, et al. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765. - PMC - PubMed
    1. Sorace J, Wong H-H, DeLeire T, et al. Quantifying the competitiveness of the electronic health record market and its implications for interoperability. Int J Med Inform 2020; 136: 104037. - PubMed
    1. Clinical Decision Support (CDS) | HealthIT.gov. https://www.healthit.gov/test-method/clinical-decision-support-cds. Accessed September 7, 2021.
    1. Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems. Ann Intern Med 2012; 157 (1): 29–43. - PubMed
    1. Mitchell J, Probst J, Brock-Martin A, et al. Association between clinical decision support system use and rural quality disparities in the treatment of pneumonia. J Rural Health 2014; 30 (2): 186–95. - PubMed

Publication types