Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 Apr 2;10(4):e0122239.
doi: 10.1371/journal.pone.0122239. eCollection 2015.

Measuring the outcome of biomedical research: a systematic literature review

Frédérique Thonon  1 , Rym Boulkedid  2 , Tristan Delory  3 , Sophie Rousseau  4 , Mahasti Saghatchian  5 , Wim van Harten  6 , Claire O'Neill  7 , Corinne Alberti  8
Affiliations

Measuring the outcome of biomedical research: a systematic literature review

Frédérique Thonon et al. PLoS One. .

Abstract

Background: There is an increasing need to evaluate the production and impact of medical research produced by institutions. Many indicators exist, yet we do not have enough information about their relevance. The objective of this systematic review was (1) to identify all the indicators that could be used to measure the output and outcome of medical research carried out in institutions and (2) enlist their methodology, use, positive and negative points.

Methodology: We have searched 3 databases (Pubmed, Scopus, Web of Science) using the following keywords: [Research outcome* OR research output* OR bibliometric* OR scientometric* OR scientific production] AND [indicator* OR index* OR evaluation OR metrics]. We included articles presenting, discussing or evaluating indicators measuring the scientific production of an institution. The search was conducted by two independent authors using a standardised data extraction form. For each indicator we extracted its definition, calculation, its rationale and its positive and negative points. In order to reduce bias, data extraction and analysis was performed by two independent authors.

Findings: We included 76 articles. A total of 57 indicators were identified. We have classified those indicators into 6 categories: 9 indicators of research activity, 24 indicators of scientific production and impact, 5 indicators of collaboration, 7 indicators of industrial production, 4 indicators of dissemination, 8 indicators of health service impact. The most widely discussed and described is the h-index with 31 articles discussing it.

Discussion: The majority of indicators found are bibliometric indicators of scientific production and impact. Several indicators have been developed to improve the h-index. This indicator has also inspired the creation of two indicators to measure industrial production and collaboration. Several articles propose indicators measuring research impact without detailing a methodology for calculating them. Many bibliometric indicators identified have been created but have not been used or further discussed.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Fig 1
Fig 1. PRISMA flowchart.
See this image and copyright information in PMC

References

    1. Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003. Jul 1;8(3):165–70 - PubMed
    1. Wooding S, Hanney S, Buxton M, Grant J. Payback arising from research funding: evaluation of the Arthritis Research Campaign. Rheumatology. 2005. Sep 1;44(9):1145–56. - PubMed
    1. Panel on return on investment in Health Research Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Canadian Academy of Health Sciences, Ottawa, ON, Canada: 2009. Available: http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf.
    1. Battersby J. Translating policy into indicators and targets In: Pencheon D, Guest C, Melzer D, Muir Gray JA, editors. Oxford Handbook of Public Health Practice- second edition Oxford: Oxford University Press; 2006. Pp.334–339.
    1. Adams J. The use of bibliometrics to measure research quality in UK higher education institutions. Arch Immunol Ther Exp (Warsz). 2009. Feb;57(1):19–32. 10.1007/s00005-009-0003-3 - DOI - PubMed

Publication types

LinkOut - more resources