Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Aug 2;40(8):btae469.
doi: 10.1093/bioinformatics/btae469.

Best practices to evaluate the impact of biomedical research software-metric collection beyond citations

Awan Afiaz  1   2 , Andrey A Ivanov  3 , John Chamberlin  4 , David Hanauer  5 , Candace L Savonen  2 , Mary J Goldman  6 , Martin Morgan  7 , Michael Reich  8 , Alexander Getka  9 , Aaron Holmes  10   11   12   13 , Sarthak Pati  9   14   15 , Dan Knight  10   11   12   13 , Paul C Boutros  10   11   12   13 , Spyridon Bakas  9   14   15 , J Gregory Caporaso  16 , Guilherme Del Fiol  4 , Harry Hochheiser  17 , Brian Haas  18 , Patrick D Schloss  19 , James A Eddy  20 , Jake Albrecht  20 , Andrey Fedorov  21 , Levi Waldron  22 , Ava M Hoffman  2 , Richard L Bradshaw  4 , Jeffrey T Leek  2 , Carrie Wright  2
Affiliations

Best practices to evaluate the impact of biomedical research software-metric collection beyond citations

Awan Afiaz et al. Bioinformatics. .

Abstract

Motivation: Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation.

Results: To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software's impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress.

Availability and implementation: More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.

PubMed Disclaimer

Conflict of interest statement

P.C.B. sits on the Scientific Advisory Boards of Sage Bionetworks, Intersect Diagnostics Inc. and BioSymetrics Inc. M.J.G. and J.G.C. act as consultants for the Fred Hutch as part of an initiative called the ITCR Training Network (www.itcrtraining.org). C.W. and C.L.S. receive royalties for some of the training materials they have written for the ITCR Training Network (www.itcrtraining.org).

Figures

Figure 1.
Figure 1.
Aspects of software infrastructure appear to be associated with a larger number of published manuscripts from users describing usage of the software in the SoftwareKG-PMC database. The X-axis indicates the age of the software by showing the year that it was released. The Y-axis indicates the log of the total number of papers that describe usage of the software in the SoftwareKG-PMC database. See Supplementary material and our website for more information.
See this image and copyright information in PMC

References

    1. Aksoy BA, Dančík V, Smith K. et al. CTD2 dashboard: a searchable web interface to connect validated results from the Cancer Target Discovery and Development Network. Database (Oxford) 2017;2017:bax054. 10.1093/database/bax054 - DOI - PMC - PubMed
    1. Amorim RC, Aguiar Castro J, Silva JRD. et al. A comparative study of platforms for research data management: interoperability, metadata capabilities and integration potential. In: R Alvaro, C Ana Maria, C Sandra, R Luis Paulo (eds.), New Contributions in Information Systems and Technologies, Advances in Intelligent Systems and Computing, Vol. 353. Cham: Springer International Publishing, 2015, 101–111. 10.1007/978-3-319-16486-1_10 - DOI
    1. Barker M, Chue Hong NP, van Eijnatten J. et al. Amsterdam Declaration on Funding Research Software Sustainability. Zenodo, 2023. 10.5281/zenodo.7740084 - DOI
    1. Basili VR, Caldiera G, Rombach DH.. The Goal Question Metric Approach, Volume I. United Kingdom: John Wiley & Sons, 1994.
    1. Begany GM, Martin EG, Yuan XJ.. Open government data portals: predictors of site engagement among early users of health data NY. Gov Inform Quart 2021;38:101614.

Publication types