Ensemble pretrained language models to extract biomedical knowledge from literature

Abstract

Objectives: The rapid expansion of biomedical literature necessitates automated techniques to discern relationships between biomedical concepts from extensive free text. Such techniques facilitate the development of detailed knowledge bases and highlight research deficiencies. The LitCoin Natural Language Processing (NLP) challenge, organized by the National Center for Advancing Translational Science, aims to evaluate such potential and provides a manually annotated corpus for methodology development and benchmarking.

Materials and methods: For the named entity recognition (NER) task, we utilized ensemble learning to merge predictions from three domain-specific models, namely BioBERT, PubMedBERT, and BioM-ELECTRA, devised a rule-driven detection method for cell line and taxonomy names and annotated 70 more abstracts as additional corpus. We further finetuned the T0pp model, with 11 billion parameters, to boost the performance on relation extraction and leveraged entites' location information (eg, title, background) to enhance novelty prediction performance in relation extraction (RE).

Results: Our pioneering NLP system designed for this challenge secured first place in Phase I-NER and second place in Phase II-relation extraction and novelty prediction, outpacing over 200 teams. We tested OpenAI ChatGPT 3.5 and ChatGPT 4 in a Zero-Shot setting using the same test set, revealing that our finetuned model considerably surpasses these broad-spectrum large language models.

Discussion and conclusion: Our outcomes depict a robust NLP system excelling in NER and RE across various biomedical entities, emphasizing that task-specific models remain superior to generic large ones. Such insights are valuable for endeavors like knowledge graph development and hypothesis formulation in biomedical research.

Keywords: ensemble learning; knowledge base; large language model; named entity recognition; relation extraction.

Figure 1.

(A) An annotated abstract with six entity types. (B) The number of mentions for each entity type in 400 abstracts for model development.

Figure 2.

The NLP system for Phase I NER task. (A) The overall architecture of the system. Ensemble learning will first combine the identified mentions from context- and fold-level, and then extract the overlapped predictions for different models as the final prediction. $S_i$ is the set of identified mentions from the model in each context, fold, or pretrained language model. $C$ is the number of contexts for each sentence in the abstract, $F$ is the number of folds, and $M$ is the number of pretrained language models we used in this study. (B) An illustration of the cross-sentence scheme for input sequence generation. (C) An illustration of the ensemble learning at context-, fold-, and model-level.

Figure 3.

An example of generating input text for T0pp training and testing using prompt template. Injecting the meta information of each candidate pair (A) into the prompt template (B), the input text (C) was generated to finetune T0pp to generate/predict the relation types of a given candidate pair in the test set.