DUVEL: an active-learning annotated biomedical corpus for the recognition of oligogenic combinations

Abstract

While biomedical relation extraction (bioRE) datasets have been instrumental in the development of methods to support biocuration of single variants from texts, no datasets are currently available for the extraction of digenic or even oligogenic variant relations, despite the reports in literature that epistatic effects between combinations of variants in different loci (or genes) are important to understand disease etiologies. This work presents the creation of a unique dataset of oligogenic variant combinations, geared to train tools to help in the curation of scientific literature. To overcome the hurdles associated with the number of unlabelled instances and the cost of expertise, active learning (AL) was used to optimize the annotation, thus getting assistance in finding the most informative subset of samples to label. By pre-annotating 85 full-text articles containing the relevant relations from the Oligogenic Diseases Database (OLIDA) with PubTator, text fragments featuring potential digenic variant combinations, i.e. gene-variant-gene-variant, were extracted. The resulting fragments of texts were annotated with ALAMBIC, an AL-based annotation platform. The resulting dataset, called DUVEL, is used to fine-tune four state-of-the-art biomedical language models: BiomedBERT, BiomedBERT-large, BioLinkBERT and BioM-BERT. More than 500 000 text fragments were considered for annotation, finally resulting in a dataset with 8442 fragments, 794 of them being positive instances, covering 95% of the original annotated articles. When applied to gene-variant pair detection, BiomedBERT-large achieves the highest F1 score (0.84) after fine-tuning, demonstrating significant improvement compared to the non-fine-tuned model, underlining the relevance of the DUVEL dataset. This study shows how AL may play an important role in the creation of bioRE dataset relevant for biomedical curation applications. DUVEL provides a unique biomedical corpus focusing on 4-ary relations between two genes and two variants. It is made freely available for research on GitHub and Hugging Face. Database URL: https://huggingface.co/datasets/cnachteg/duvel or https://doi.org/10.57967/hf/1571.

Figure 1.

Schematic representation of the whole annotation process. Unlabelled instances are represented as unfilled circles, labelled instances as filled circles and the machine learning model as gears. The left picture depicts the filtering process of the OLIDAv2 articles, starting with 318 articles, to the 85 articles at the origin of the instances composing the DUVEL dataset. The right side represents the annotation with AL, with 1000 and 500 instances randomly selected from all the 511 635 unlabelled instances for the test set and initial labelled set, respectively. The labelled set is used to fine-tune the BiomedBERT model, whose performance is evaluated with the test set. The fine-tuned model selects 500 instances in the unlabelled instances, which it considers as the most informative according to the Margin selection strategy. The selected instances are then labelled and added to the labelled set for a new iteration of fine-tuning-selection-labelling. The process stops once 4500 instances have been selected and labelled.

Figure 2.

Examples of the different classes. Genes and variants are highlighted in the text. (A) Example of a text for the negative class. In this example, the variants and genes belong to two different patients and thus are not involved in a digenic variant relation. (B) Example of a text for the positive class. The text clearly states that the gene–variant pair is co-inherited in patients. The digenic variant combination depicted here is OLI504, more information can be found on https://olida.ibsquare.be/detail/Combination/OLI504/.

Figure 3.

Fraction of positive samples in the training set across the AL iterations. The initial randomly selected labelled set has a positive fraction of 1%, which increases to more than 14% after the first round of AL selection and the addition of the 500 samples. The fraction of positive instances decreases slightly during the next two rounds of AL, but after that reaches and maintains a positive fraction between 13% and 15% until the end of the AL process.

Figure 4.

F1-score over the AL iterations during the annotation of the DUVEL data set, evaluated with the DUVEL test set instead of the test set initialised during the AL process. The initially selected samples during the AL process are filtered to exclude the samples present in the DUVEL test set.