Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Oct 3:26:e60601.
doi: 10.2196/60601.

Ascle-A Python Natural Language Processing Toolkit for Medical Text Generation: Development and Evaluation Study

Rui Yang #  1 , Qingcheng Zeng #  2 , Keen You #  3 , Yujie Qiao #  4 , Lucas Huang  3 , Chia-Chun Hsieh  3 , Benjamin Rosand  3 , Jeremy Goldwasser  3 , Amisha Dave  5 , Tiarnan Keenan  6 , Yuhe Ke  7 , Chuan Hong  8 , Nan Liu  1   9   10 , Emily Chew  6 , Dragomir Radev  3 , Zhiyong Lu  11 , Hua Xu  12 , Qingyu Chen  12 , Irene Li  13   14
Affiliations

Ascle-A Python Natural Language Processing Toolkit for Medical Text Generation: Development and Evaluation Study

Rui Yang et al. J Med Internet Res. .

Abstract

Background: Medical texts present significant domain-specific challenges, and manually curating these texts is a time-consuming and labor-intensive process. To address this, natural language processing (NLP) algorithms have been developed to automate text processing. In the biomedical field, various toolkits for text processing exist, which have greatly improved the efficiency of handling unstructured text. However, these existing toolkits tend to emphasize different perspectives, and none of them offer generation capabilities, leaving a significant gap in the current offerings.

Objective: This study aims to describe the development and preliminary evaluation of Ascle. Ascle is tailored for biomedical researchers and clinical staff with an easy-to-use, all-in-one solution that requires minimal programming expertise. For the first time, Ascle provides 4 advanced and challenging generative functions: question-answering, text summarization, text simplification, and machine translation. In addition, Ascle integrates 12 essential NLP functions, along with query and search capabilities for clinical databases.

Methods: We fine-tuned 32 domain-specific language models and evaluated them thoroughly on 27 established benchmarks. In addition, for the question-answering task, we developed a retrieval-augmented generation (RAG) framework for large language models that incorporated a medical knowledge graph with ranking techniques to enhance the reliability of generated answers. Additionally, we conducted a physician validation to assess the quality of generated content beyond automated metrics.

Results: The fine-tuned models and RAG framework consistently enhanced text generation tasks. For example, the fine-tuned models improved the machine translation task by 20.27 in terms of BLEU score. In the question-answering task, the RAG framework raised the ROUGE-L score by 18% over the vanilla models. Physician validation of generated answers showed high scores for readability (4.95/5) and relevancy (4.43/5), with a lower score for accuracy (3.90/5) and completeness (3.31/5).

Conclusions: This study introduces the development and evaluation of Ascle, a user-friendly NLP toolkit designed for medical text generation. All code is publicly available through the Ascle GitHub repository. All fine-tuned language models can be accessed through Hugging Face.

Keywords: deep learning; generative artificial intelligence; healthcare; large language models; machine learning; natural language processing; retrieval-augmented generation.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
The overall architecture of Ascle. formula image indicates that we have our fine-tuned language models for this task. formula image indicates that we conducted evaluations for this task. POS: Parts-Of-Speech; QA: Question-Answering; UMLS: Unified Medical Language System.
Figure 2
Figure 2
(A) Physician validation (readability, relevancy, accuracy, and completeness) for 50 question-answer pairs. (B) Two examples of generated answers with ground truth.
Figure 3
Figure 3
Demonstration of system usage. We show two use cases: Text Simplification and Machine Translation.
See this image and copyright information in PMC

Update of

References

    1. Li I, Yasunaga M, Nuzumlalı MY, Caraballo C, Mahajan S, Krumholz H, Radev D. A neural topic-attention model for medical term abbreviation disambiguation. ArXiv. Preprint posted online on October 30, 2019. 2019 doi: 10.5260/chara.21.2.8. https://arxiv.org/abs/1910.14076 - DOI
    1. Li I, Pan J, Goldwasser J, Verma N, Wong WP, Nuzumlalı MY, Rosand B, Li Y, Zhang M, Chang D, Taylor RA, Krumholz HM, Radev D. Neural natural language processing for unstructured data in electronic health records: a review. Computer Science Review. 2022;46:100511. doi: 10.1016/j.cosrev.2022.100511. - DOI
    1. Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2018;22(5):1589–1604. doi: 10.1109/JBHI.2017.2767063. https://europepmc.org/abstract/MED/29989977 - DOI - PMC - PubMed
    1. al-Aiad A, Duwairi R, Fraihat M. Survey: deep learning concepts and techniques for electronic health record. IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA); 2018 October 28 - 2018 November 01; Aqaba, Jordan. IEEE; 2018. pp. 1–5. - DOI
    1. Zhang Y, Chen Q, Yang Z, Lin H, Lu Z. BioWordVec, improving biomedical word embeddings with subword information and MeSH. Sci Data. 2019;6(1):52. doi: 10.1038/s41597-019-0055-0. http://europepmc.org/abstract/MED/31076572 10.1038/s41597-019-0055-0 - DOI - PMC - PubMed

Publication types

LinkOut - more resources