The effect of using a large language model to respond to patient messages

Shan Chen¹, Marco Guevara², Shalini Moningi³, Frank Hoebers⁴, Hesham Elhalawani³, Benjamin H Kann², Fallon E Chipidza³, Jonathan Leeman³, Hugo J W L Aerts⁵, Timothy Miller⁶, Guergana K Savova⁶, Jack Gallifant⁷, Leo A Celi⁸, Raymond H Mak², Maryam Lustberg⁹, Majid Afshar¹⁰, Danielle S Bitterman¹¹

Affiliations

¹ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA.
² Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA.
³ Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA.
⁴ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Department of Radiation Oncology, GROW School for Oncology and Reproduction, Maastricht University, Maastricht, Netherlands.
⁵ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Radiology and Nuclear Medicine, GROW and Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, Netherlands.
⁶ Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA.
⁷ Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA.
⁸ Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA; Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA.
⁹ Department of Medical Oncology, Yale School of Medicine, New Haven, CT, USA.
¹⁰ Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
¹¹ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA. Electronic address: dbitterman@bwh.harvard.edu.

PMID: 38664108
PMCID: PMC11829255
DOI: 10.1016/S2589-7500(24)00060-8

The effect of using a large language model to respond to patient messages

Shan Chen et al. Lancet Digit Health. 2024 Jun.

. 2024 Jun;6(6):e379-e381.

doi: 10.1016/S2589-7500(24)00060-8. Epub 2024 Apr 24.

Authors

Affiliations

¹ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA.
² Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA.
³ Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA.
⁴ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Department of Radiation Oncology, GROW School for Oncology and Reproduction, Maastricht University, Maastricht, Netherlands.
⁵ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Radiology and Nuclear Medicine, GROW and Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, Netherlands.
⁶ Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA.
⁷ Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA.
⁸ Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA; Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA.
⁹ Department of Medical Oncology, Yale School of Medicine, New Haven, CT, USA.
¹⁰ Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
¹¹ Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA. Electronic address: dbitterman@bwh.harvard.edu.

PMID: 38664108
PMCID: PMC11829255
DOI: 10.1016/S2589-7500(24)00060-8

No abstract available

PubMed Disclaimer

Conflict of interest statement

DSB reports being an Associate Editor of Radiation Oncology at HemOnc.org (no financial compensation, unrelated to this work, and recieving funding from American Association for Cancer Research, unrelated to this work. HJWLA reports advising and consulting for Onc.AI, Love Health, Sphera, Editas, AstraZeneca, and Bristol Myers Squibb, unrelated to this work. RHM reports being on an Advisory Board for ViewRay and AstraZeneca; Consulting for Varian Medical Systems and Sio Capital Management; and honorarium from Novartis and Springer Nature. JL reports research funding from Viewray, NH Theraguix, and Varian. ML reports advisory and consulting for Pfizer, Gilead, Novartis, and AstraZeneca, unrelated to this work. BHK reports research funding from Botha-Chan Low Grade Glioma Consortium (National institutes of Health [NIH]-USA K08DE030216-01). All other authors declare no competing interests. The authors acknowledge financial support from the Woods Foundation (DSB, RHM, BHK, and HJWLA) NIH (NIH-USA U54CA274516-01A1 (SC, MG, BHK, HJWLA, GKS, and DSB), NIH-USA U24CA194354 (HJWLA), NIH-USA U01CA190234 (HJWLA), NIH-USA U01CA209414 (HJWLA), and NIH-USA R35CA22052 (HJWLA), NIH-NIDA R01DA051464 (MA), R01GM114355 (GKS), NIH-USA R01LM012973 (TM and MA), NIH-USA R01MH126977 (TM), NIH-USA U54 TW012043-01 (JG and LAC), NIH-USA OT2OD032701 (JG and LAC), NIH-USA R01EB017205 (LAC), and the EU European Research Council (HJWLA 866504), all outside of the submitted work. All data collected and generated in this study, after de-identification, are available at https://github.com/AIM-Harvard/OncQA. SC: conceptualisation, data curation, formal analysis, investigation, methodology, visualisation, and writing (original draft, review, and editing). MG: conceptualisation, data curation, and formal analysis. SM, FH EH, BHK, FEC, JL: data curation, investigation, and methodology. RHM: data curation, investigation, methodology, and writing (review and editing). HJWLA: investigation, methodology, resources, and writing (review and editing). JG: formal analysis, investigation, methodology, visualisation, and writing (review and editing). TM and GKS: formal analysis, investigation, methodology, and writing (review and editing). ML data curation, formal analysis, investigation, and methodology. LAC formal analysis, investigation, supervision, and writing (review and editing). MA: conceptualisation, data curation, formal analysis, investigation, methodology, supervision, and writing (review and editing). DSB: conceptualisation, data curation, formal analysis, investigation, methodology, supervision, visualisation, resources, and writing (original draft, review, and editing). SC and DSB directly accessed and verified the underlying data reported in the manuscript. All authors have full access to all the data in the study and accept responsibility to submit for publication.

Figures

See this image and copyright information in PMC

Comment in

From text to treatment: the crucial role of validation for generative large language models in health care.
de Hond A, Leeuwenberg T, Bartels R, van Buchem M, Kant I, Moons KG, van Smeden M. de Hond A, et al. Lancet Digit Health. 2024 Jul;6(7):e441-e443. doi: 10.1016/S2589-7500(24)00111-0. Lancet Digit Health. 2024. PMID: 38906607 No abstract available.

References

1. Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020; 27: 531–38. - PMC - PubMed
1. Epic. Epic and Microsoft bring GPT-4 to EHRs. May 5, 2023. https://www.epic.com/epic/post/epic-and-microsoft-bring-gpt-4-to-ehrs (accessed March 1, 2024).
1. Nath B, Williams B, Jeffery MM, et al. Trends in electronic health record inbox messaging during the COVID-19 pandemic in an ambulatory practice network in New England. JAMA Netw Open 2021; 4: e2131490. - PMC - PubMed
1. Chen S, Kann BH, Foote MB, et al. Use of artificial intelligence chatbots for cancer treatment information. JAMA Oncol 2023; 9: 1459–62. - PMC - PubMed
1. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature 2023; 620: 172–80. - PMC - PubMed

Grants and funding

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

The effect of using a large language model to respond to patient messages

Affiliations

The effect of using a large language model to respond to patient messages

Authors

Affiliations

Conflict of interest statement

Figures

Comment in

References

Grants and funding

LinkOut - more resources

Full Text Sources