Using text-mining to measure the scientific impact and legacy of ELIXIR, a distributed research infrastructure for life science data

Abstract

Background: ELIXIR is a pan-European public-funded research infrastructure dedicated to life science data. As such, it must demonstrate public value to its funders and stakeholders. We present methods to inventory research publications linked to ELIXIR that have received funding and support, as well as related citation metrics, used as performance metrics for these audiences.

Methods: To overcome challenges inherent in ELIXIR's distributed structure, and the fact that those publishing ELIXIR-supported work are typically working part-time on ELIXIR matters, a semi-automated approach, consisting of text-mining followed by manual curation, is presented. A country-level case study (ELIXIR Italy) refines and expands the methods, notably by introducing more granularity in the curation process (e.g. considering all national-level grants, examining affiliations to report publication per institute) and by additionally looking at the scientific impact of the resources developed and operated by the Italian Node of ELIXIR.

Results: Overall, the methods described in this article have shown to: (1) be repeatable with acceptable levels of accuracy and consistency (notably across curators); (2) require reasonable effort in terms of curation of monthly 'harvests' of publications (obtained by text-mining); and (3) to be well-adapted to ELIXIR's distributed nature.

Conclusions: Concrete examples are provided of downstream uses of the inventoried publications and their citations, both for ELIXIR as a whole and for the Italian case study. Limitations of the methods are discussed, particularly the challenges associated with using an 'Open literature' database (Europe PMC) for the text-mining, and the constraints related to curation capacity. The methods, along with the valuable lessons learned during their development, are sufficiently generic and pragmatic enough to be readily adapted by other similar research infrastructures.

Keywords: KPI; bioinformatics; database; funder; literature; metric; performance; resource.

Figure 1.. Publications and citations supported by ELIXIR (2011–2023) in Open literature (EuropePMC).

Figure 2.. Publications and citations supported by ELIXIR Italy (2011–2023) in Open literature (EuropePMC).

Figure 3.. Distribution of ELIXIR Italy publications (double counted) across its Node institutes.

Acronyms: see https://elixir-italy.org/about/members/.