Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Jun 12:7:1588143.
doi: 10.3389/fdgth.2025.1588143. eCollection 2025.

AI-generated draft replies to patient messages: exploring effects of implementation

Charlotte M H H T Bootsma-Robroeks  1   2 , Jessica D Workum  3   4 , Stephanie C E Schuit  5 , Anne Hoekman  1 , Tarannom Mehri  1 , Job N Doornberg  6   7 , Tom P van der Laan  1   8 , Rosanne C Schoonbeek  1   8
Affiliations

AI-generated draft replies to patient messages: exploring effects of implementation

Charlotte M H H T Bootsma-Robroeks et al. Front Digit Health. .

Abstract

Introduction: The integration of Large Language Models (LLMs) in Electronic Health Records (EHRs) has the potential to reduce administrative burden. Validating these tools in real-world clinical settings is essential for responsible implementation. In this study, the effect of implementing LLM-generated draft responses to patient questions in our EHR is evaluated with regard to adoption, use and potential time savings.

Material and methods: Physicians across 14 medical specialties in a non-English large academic hospital were invited to use LLM-generated draft replies during this prospective observational clinical cohort study of 16 weeks, choosing either the drafted or a blank reply. The adoption rate, the level of adjustments to the initial drafted responses compared to the final sent messages (using ROUGE-1 and BLEU-1 natural language processing scores), and the time spent on these adjustments were analyzed.

Results: A total of 919 messages by 100 physicians were evaluated. Clinicians used the LLM draft in 58% of replies. Of these, 43% used a large part of the suggested text for the final answer (≥10% match drafted responses: ROUGE-1: 86% similarity, vs. blank replies: ROUGE-1: 16%). Total response time did not significantly different when using a blank reply compared to using a drafted reply with ≥10% match (157 vs. 153 s, p = 0.69).

Discussion: General adoption of LLM-generated draft responses to patient messages was 58%, although the level of adjustments on the drafted message varied widely between medical specialties. This implicates safe use in a non-English, tertiary setting. The current implementation has not yet resulted in time savings, but a learning curve can be expected.

Registration number: 19035.

Keywords: LLM generated draft responses; adoption; electronic health records; inbasket messages; large language model (LLM); time saving.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1
Figure 1
Flow of incoming patient messages, LLM integration, and the generation of draft replies.
Figure 2
Figure 2
Flowchart of messages answered with draft replies.
See this image and copyright information in PMC

References

    1. Akbar F, Mark G, Warton EM, Reed ME, Prausnitz S, East JA, et al. Physicians’ electronic inbox work patterns and factors associated with high inbox work duration. J Am Med Inform Assoc. (2021) 28(5):923–30. 10.1093/jamia/ocaa229 - DOI - PMC - PubMed
    1. Baratta LR, Harford D, Sinsky CA, Kannampallil T, Lou SS. Characterizing the patterns of electronic health record-integrated secure messaging use: cross-sectional study. J Med Internet Res. (2023) 25(1):e48583. 10.2196/48583 - DOI - PMC - PubMed
    1. Rao SK, Kimball AB, Lehrhoff SR, Hidrue MK, Colton DG, Ferris TG, et al. The impact of administrative burden on academic physicians: results of a hospital-wide physician survey. Acad Med. (2017) 92(2):237–43. 10.1097/ACM.0000000000001461 - DOI - PubMed
    1. Holmgren AJ, Downing NL, Tang M, Sharp C, Longhurst C, Huckman RS. Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. J Am Med Inform Assoc. (2022) 29(3):453–60. 10.1093/jamia/ocab268 - DOI - PMC - PubMed
    1. Baumann LA, Baker J, Elshaug AG. The impact of electronic health record systems on clinical documentation times: a systematic review. Health Policy. (2018) 122(8):827–36. 10.1016/j.healthpol.2018.05.014 - DOI - PubMed

LinkOut - more resources