Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Mar 29;40(4):btae163.
doi: 10.1093/bioinformatics/btae163.

Advancing entity recognition in biomedicine via instruction tuning of large language models

Vipina K Keloth  1 , Yan Hu  2 , Qianqian Xie  1 , Xueqing Peng  1 , Yan Wang  1 , Andrew Zheng  3 , Melih Selek  4 , Kalpana Raja  1 , Chih Hsuan Wei  5 , Qiao Jin  5 , Zhiyong Lu  5 , Qingyu Chen  1   5 , Hua Xu  1
Affiliations

Advancing entity recognition in biomedicine via instruction tuning of large language models

Vipina K Keloth et al. Bioinformatics. .

Abstract

Motivation: Large Language Models (LLMs) have the potential to revolutionize the field of Natural Language Processing, excelling not only in text generation and reasoning tasks but also in their ability for zero/few-shot learning, swiftly adapting to new tasks with minimal fine-tuning. LLMs have also demonstrated great promise in biomedical and healthcare applications. However, when it comes to Named Entity Recognition (NER), particularly within the biomedical domain, LLMs fall short of the effectiveness exhibited by fine-tuned domain-specific models. One key reason is that NER is typically conceptualized as a sequence labeling task, whereas LLMs are optimized for text generation and reasoning tasks.

Results: We developed an instruction-based learning paradigm that transforms biomedical NER from a sequence labeling task into a generation task. This paradigm is end-to-end and streamlines the training and evaluation process by automatically repurposing pre-existing biomedical NER datasets. We further developed BioNER-LLaMA using the proposed paradigm with LLaMA-7B as the foundational LLM. We conducted extensive testing on BioNER-LLaMA across three widely recognized biomedical NER datasets, consisting of entities related to diseases, chemicals, and genes. The results revealed that BioNER-LLaMA consistently achieved higher F1-scores ranging from 5% to 30% compared to the few-shot learning capabilities of GPT-4 on datasets with different biomedical entities. We show that a general-domain LLM can match the performance of rigorously fine-tuned PubMedBERT models and PMC-LLaMA, biomedical-specific language model. Our findings underscore the potential of our proposed paradigm in developing general-domain LLMs that can rival SOTA performances in multi-task, multi-domain scenarios in biomedical and health applications.

Availability and implementation: Datasets and other resources are available at https://github.com/BIDS-Xu-Lab/BioNER-LLaMA.

PubMed Disclaimer

Conflict of interest statement

The authors do not have any conflicts of interest to disclose.

Figures

Figure 1.
Figure 1.
Framework for the development of instruction tuned large language models utilizing existing NER datasets.
Figure 2.
Figure 2.
A prompt example used for instruction tuning to extract a disease mention (constructed from NCBI disease dataset).
Figure 3.
Figure 3.
Error analysis on 100 samples of BioNER-LLaMA predictions
Figure 4.
Figure 4.
Precision, recall, and F1. Variations in values when the size of training dataset is reduced to 10%, 25%, and 50% compared to full training data for BioNER-LLaMA
See this image and copyright information in PMC

References

    1. Achiam J, Adler S, Agarwal S. et al. Gpt-4 technical report. arXiv, arXiv:2303.08774, 2023, preprint: not peer reviewed.
    1. Agrawal M, Hegselmann S, Lang H. et al. Large language models are few-shot clinical information extractors. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, UAE. 2022.
    1. Ashok D, Lipton ZC. PromptNER: prompting for named entity recognition. arXiv, arXiv:2305.15444, 2023, preprint: not peer reviewed.
    1. Beltagy I, Lo K, Cohan A. SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620, Hong Kong, China. Association for Computational Linguistics. 2019.
    1. Biderman S, Schoelkopf H, Anthony QG. et al. Pythia: a suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning, Honolulu, Hawaii, USA. 2023.

Publication types