Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and Format

Abstract

Importance: By law, patients have immediate access to discharge notes in their medical records. Technical language and abbreviations make notes difficult to read and understand for a typical patient. Large language models (LLMs [eg, GPT-4]) have the potential to transform these notes into patient-friendly language and format.

Objective: To determine whether an LLM can transform discharge summaries into a format that is more readable and understandable.

Design, setting, and participants: This cross-sectional study evaluated a sample of the discharge summaries of adult patients discharged from the General Internal Medicine service at NYU (New York University) Langone Health from June 1 to 30, 2023. Patients discharged as deceased were excluded. All discharge summaries were processed by the LLM between July 26 and August 5, 2023.

Interventions: A secure Health Insurance Portability and Accountability Act-compliant platform, Microsoft Azure OpenAI, was used to transform these discharge summaries into a patient-friendly format between July 26 and August 5, 2023.

Main outcomes and measures: Outcomes included readability as measured by Flesch-Kincaid Grade Level and understandability using Patient Education Materials Assessment Tool (PEMAT) scores. Readability and understandability of the original discharge summaries were compared with the transformed, patient-friendly discharge summaries created through the LLM. As balancing metrics, accuracy and completeness of the patient-friendly version were measured.

Results: Discharge summaries of 50 patients (31 female [62.0%] and 19 male [38.0%]) were included. The median patient age was 65.5 (IQR, 59.0-77.5) years. Mean (SD) Flesch-Kincaid Grade Level was significantly lower in the patient-friendly discharge summaries (6.2 [0.5] vs 11.0 [1.5]; P < .001). PEMAT understandability scores were significantly higher for patient-friendly discharge summaries (81% vs 13%; P < .001). Two physicians reviewed each patient-friendly discharge summary for accuracy on a 6-point scale, with 54 of 100 reviews (54.0%) giving the best possible rating of 6. Summaries were rated entirely complete in 56 reviews (56.0%). Eighteen reviews noted safety concerns, mostly involving omissions, but also several inaccurate statements (termed hallucinations).

Conclusions and relevance: The findings of this cross-sectional study of 50 discharge summaries suggest that LLMs can be used to translate discharge summaries into patient-friendly language and formats that are significantly more readable and understandable than discharge summaries as they appear in electronic health records. However, implementation will require improvements in accuracy, completeness, and safety. Given the safety concerns, initial implementation will require physician review.

Figure 1.. Example of a Patient-Friendly Discharge Summary Created Using a Large Language Model

This example was created from a hypothetical discharge summary. ECG indicates electrocardiogram; STEMI, ST-segment elevation myocardial infarction.

Figure 2.. Readability of Discharge Summaries

Box plot compares total word count, Flesch-Kincaid (FK) Reading Ease score, and FK Grade Level. Blue boxes represent the original discharge summary; orange boxes represent the patient-friendly discharge summary. For FK Reading Ease score, higher scores indicate better outcomes; for FK Grade Level, lower grade levels indicate better outcomes. Vertical lines indicate means; box, IQRs; dots, outliers; and error bars, minimum and maximum excluding outliers.

Figure 3.. Counts of Accuracy Ratings

The counts of accuracy ratings on a scale of 1 to 6 from each of 100 reviewers, where 1 indicates completely incorrect; 2, more incorrect than correct; 3, approximately equally correct and incorrect; 4, more correct than incorrect; 5, nearly all correct; and 6, completely correct.