Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Aug 28;5(1):376.
doi: 10.1038/s43856-025-01091-3.

Development and evaluation of a clinical note summarization system using large language models

Juliana Damasio Oliveira #  1 , Henrique D P Santos #  2 , Ana Helena D P S Ulbrich #  2 , Julia Colleoni Couto #  2 , Marcelo Arocha #  2 , Joaquim Santos #  2 , Manuela Martins Costa  2 , Daniela Faccio  2 , Fabio O Tabalipa  3 , Rodrigo F Nogueira  4
Affiliations

Development and evaluation of a clinical note summarization system using large language models

Juliana Damasio Oliveira et al. Commun Med (Lond). .

Abstract

Background: Clinical notes are a vital and detailed source of information about patient hospitalizations. However, the sheer volume and complexity of these notes make evaluation and summarization challenging. Nonetheless, summarizing clinical notes is essential for accurate and efficient clinical decision-making in patient care. Generative language models, particularly large language models such as GPT-4, offer a promising solution by creating coherent, contextually relevant text based on patterns learned from large datasets.

Methods: This study describes the development of a discharge summary system using large language models. By conducting an online survey and interviews, we gather feedback from end users, including physicians and patients, to ensure the system meets their practical needs and fits their experiences. Additionally, we develop a rating system to evaluate prompt effectiveness by comparing model-generated outputs with human assessments, which serve as benchmarks to evaluate the performance of the automated model.

Results: Here we show that the model's ability to interpret diagnoses borders on humanlevel accuracy, demonstrating its potential to assist healthcare professionals in routine tasks such as generating discharge summaries.

Conclusions: This advancement underscores the potential of large language models in clinical settings and opens up possibilities for broader applications in healthcare documentation and decision-making support.

Plain language summary

This study developed a system to support physicians in writing hospital discharge summaries. Clinical notes often include essential patient information, but their length and complexity can make it challenging to summarize them efficiently. To address this, we applied artificial intelligence (AI) techniques to help generate clear and organized summaries based on patient data. We collected input from both physicians and patients through surveys and interviews to ensure the system aligned with their needs. We also evaluated the summaries created by the system by comparing them to those written by healthcare professionals. The results showed that the AI-generated summaries were comparable in accuracy to human-written versions. This suggests that such a system could assist physicians in their documentation tasks and contribute to clearer communication during care transitions. Future applications may include other types of clinical documentation.

PubMed Disclaimer

Conflict of interest statement

Competing interests: The authors declare no competing interests.

Figures

Fig. 1
Fig. 1. Process to develop and evaluate the discharge summary.
We employed a three-step approach, beginning with (1) requirements gathering, followed by (2) development, which involved an online survey and interviews. The final step focused on (3) evaluating the system we created.
Fig. 2
Fig. 2. NoHarm discharge summary—web interface.
Discharge summary generated by our AI-based system. The summary demonstrates how structured and unstructured clinical data are organized to support effective communication among healthcare professionals during patient discharge.
See this image and copyright information in PMC

References

    1. Chua, C. E. & Teo, D. B. Writing a high-quality discharge summary through structured training and assessment. Med. Educ.57, 773–774 (2023). - PubMed
    1. Sebastianus, F. & Suharto, E. Information system design completeness of filling out discharge summary of inpatients. J. Tek. Inform.3, 877–887 (2022).
    1. Dielissen, P. W. & Beuken-van Everdingen, M. Quality of discharge summary for patients with limited life expectancy. Ned. Tijdschr. Voor Geneeskd.166, 6575–6575 (2022). - PubMed
    1. Chatterton, B. et al. Primary care physicians’ perspectives on high-quality discharge summaries. J. Gen. Intern. Med.39, 1438–1443 (2024). - PMC - PubMed
    1. Goodman, H. Discharging patients from acute care hospitals. Nurs. Stand.30, 49–60 (2016). - PubMed

LinkOut - more resources