Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Randomized Controlled Trial
. 2024 Apr 1;7(4):e246565.
doi: 10.1001/jamanetworkopen.2024.6565.

AI-Generated Draft Replies Integrated Into Health Records and Physicians' Electronic Communication

Ming Tai-Seale  1   2 , Sally L Baxter  2   3 , Florin Vaida  4 , Amanda Walker  1 , Amy M Sitapati  2 , Chad Osborne  1 , Joseph Diaz  2 , Nimit Desai  5 , Sophie Webb  1 , Gregory Polston  6 , Teresa Helsten  2 , Erin Gross  7 , Jessica Thackaberry  8 , Ammar Mandvi  1 , Dustin Lillie  1 , Steve Li  2 , Geneen Gin  1 , Suraj Achar  1 , Heather Hofflich  2 , Christopher Sharp  9 , Marlene Millen  2 , Christopher A Longhurst  2   10
Affiliations
Randomized Controlled Trial

AI-Generated Draft Replies Integrated Into Health Records and Physicians' Electronic Communication

Ming Tai-Seale et al. JAMA Netw Open. .

Abstract

Importance: Timely tests are warranted to assess the association between generative artificial intelligence (GenAI) use and physicians' work efforts.

Objective: To investigate the association between GenAI-drafted replies for patient messages and physician time spent on answering messages and the length of replies.

Design, setting, and participants: Randomized waiting list quality improvement (QI) study from June to August 2023 in an academic health system. Primary care physicians were randomized to an immediate activation group and a delayed activation group. Data were analyzed from August to November 2023.

Exposure: Access to GenAI-drafted replies for patient messages.

Main outcomes and measures: Time spent (1) reading messages, (2) replying to messages, (3) length of replies, and (4) physician likelihood to recommend GenAI drafts. The a priori hypothesis was that GenAI drafts would be associated with less physician time spent reading and replying to messages. A mixed-effects model was used.

Results: Fifty-two physicians participated in this QI study, with 25 randomized to the immediate activation group and 27 randomized to the delayed activation group. A contemporary control group included 70 physicians. There were 18 female participants (72.0%) in the immediate group and 17 female participants (63.0%) in the delayed group; the median age range was 35-44 years in the immediate group and 45-54 years in the delayed group. The median (IQR) time spent reading messages in the immediate group was 26 (11-69) seconds at baseline, 31 (15-70) seconds 3 weeks after entry to the intervention, and 31 (14-70) seconds 6 weeks after entry. The delayed group's median (IQR) read time was 25 (10-67) seconds at baseline, 29 (11-77) seconds during the 3-week waiting period, and 32 (15-72) seconds 3 weeks after entry to the intervention. The contemporary control group's median (IQR) read times were 21 (9-54), 22 (9-63), and 23 (9-60) seconds in corresponding periods. The estimated association of GenAI was a 21.8% increase in read time (95% CI, 5.2% to 41.0%; P = .008), a -5.9% change in reply time (95% CI, -16.6% to 6.2%; P = .33), and a 17.9% increase in reply length (95% CI, 10.1% to 26.2%; P < .001). Participants recognized GenAI's value and suggested areas for improvement.

Conclusions and relevance: In this QI study, GenAI-drafted replies were associated with significantly increased read time, no change in reply time, significantly increased reply length, and some perceived benefits. Rigorous empirical tests are necessary to further examine GenAI's performance. Future studies should examine patient experience and compare multiple GenAIs, including those with medical training.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Baxter reported receiving nonfinancial support from Topcon and Optomed outside the submitted work. Mr Desai reported receiving personal fees from Pearl Health outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Read Time per Message (in Seconds) for Each Group and Study Period
Symbols indicate geometric means and whiskers indicated 95% CIs. T0 indicates preactivation period, 1 to 21 days before randomization day; T1, 0 to 20 days post randomization and activation (immediate activation group), or 0 to 20 days after randomization date (delayed group); T2, 21 to 41 days post activation (immediate group), or 0 to 20 days post activation (delayed group).
Figure 2.
Figure 2.. Reply Time per Message (in Seconds) for Each Group and Study Period
Symbols indicate geometric means and whiskers indicated 95% CIs. T0 indicates preactivation period, 1 to 21 days before randomization day; T1, 0 to 20 days post randomization and activation (immediate activation group), or 0 to 20 days after randomization date (delayed group); T2, 21 to 41 days post activation (immediate group), or 0 to 20 days post activation (delayed group).
Figure 3.
Figure 3.. Reply Length for Each Group and Study Period
Symbols indicate geometric means and whiskers indicated 95% CIs. T0 indicates preactivation period, 1 to 21 days before randomization day; T1, 0 to 20 days post randomization and activation (immediate activation group), or 0 to 20 days after randomization date (delayed group); T2, 21 to 41 days post activation (immediate group), or 0 to 20 days post activation (delayed group).
See this image and copyright information in PMC

References

    1. Tai-Seale M, Dillon EC, Yang Y, et al. . Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood). 2019;38(7):1073-1078. doi:10.1377/hlthaff.2018.05509 - DOI - PubMed
    1. Arndt BG, Beasley JW, Watkinson MD, et al. . Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med. 2017;15(5):419-426. doi:10.1370/afm.2121 - DOI - PMC - PubMed
    1. Akbar F, Mark G, Prausnitz S, et al. . Physician stress during electronic health record inbox work: in situ measurement with wearable sensors. JMIR Med Inform. 2021;9(4):e24014. doi:10.2196/24014 - DOI - PMC - PubMed
    1. McClellan SR, Panattoni L, Chan AS, Tai-Seale M. Patient-initiated electronic messages and quality of care for patients with diabetes and hypertension in a large fee-for-service medical group: results from a natural experiment. Med Care. 2016;54(3):287-295. doi:10.1097/MLR.0000000000000483 - DOI - PubMed
    1. Holmgren AJ, Byron ME, Grouse CK, Adler-Milstein J. Association between billing patient portal messages as e-visits and patient messaging volume. JAMA. 2023;329(4):339-342. doi:10.1001/jama.2022.24710 - DOI - PMC - PubMed

Publication types