Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 May 2;20(1):49.
doi: 10.1186/s12961-022-00856-9.

Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives

Anna-Aurora Kork  1 , Carla Antonini  2 , Nicolás García-Torea  3 , Mercedes Luque-Vílchez  4   5 , Ericka Costa  6 , Juliette Senn  7 , Carlos Larrinaga  3 , Deborah Bertorello  8 , Giampaolo Brichetto  8 , Paola Zaratin  8 , Michele Andreaus  6
Affiliations

Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives

Anna-Aurora Kork et al. Health Res Policy Syst. .

Abstract

Background: The need to more collaboratively measure the impact of health research and to do so from multidimensional perspectives has been acknowledged. A scorecard was developed as part of the Collective Research Impact Framework (CRIF), to engage stakeholders in the assessment of the impacts of health research and innovations. The purpose of this study was to describe the developmental process of the MULTI-ACT Master Scorecard (MSC) and how it can be used as a workable tool for collectively assessing future responsible research and innovation measures.

Methods: An extensive review of the health research impact literature and of multistakeholder initiatives resulted in a database of 1556 impact indicators. The MSC was then cocreated by engaging key stakeholders and conducting semi-structured interviews of experts in the field.

Results: The MSC consists of five accountability dimensions: excellence, efficacy, economic, social and patient-reported outcomes. The tool contains 125 potential indicators, classified into 53 impact measurement aspects that are considered the most relevant topics for multistakeholder research and innovation initiatives when assessing their impact on the basis of their mission and their stakeholders' interests. The scorecard allows the strategic management of multistakeholder research initiatives to demonstrate their impact on people and society. The value of the tool is that it is comprehensive, customizable and easy to use.

Conclusions: The MSC is an example of how the views of society can be taken into account when research impacts are assessed in a more sustainable and balanced way. The engagement of patients and other stakeholders is an integral part of the CRIF, facilitating collaborative decision-making in the design of policies and research agendas. In policy making, the collective approach allows the evaluation perspective to be extended to the needs of society and towards responsible research and innovation. Multidimensionality makes research and innovations more responsive to systemic challenges, and developing more equitable and sustainable health services.

Keywords: Measurement; Multistakeholder; Patient-reported dimension; Payback; Research impact; Responsible research and innovation; Scorecard.

PubMed Disclaimer

Conflict of interest statement

At the time of the study, the authors were members of the MULTI-ACT consortium and they claim to have no competing interests.

Figures

Fig. 1
Fig. 1
Interrelated impact measurement dimensions in CRIF
Fig. 2
Fig. 2
Key phases in development process of MULTI-ACT MSC
Fig. 3
Fig. 3
Utilization of MSC as part of CRIF workflow to accomplish the shared mission. The scorecard is relevant for the construction process of a shared measurement system, the co-selection of impact aspects and indicators, and for ensuring that the agenda remains in line with the mission
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2015;13(1):3. doi: 10.1186/1478-4505-13-3. - DOI - PMC - PubMed
    1. Sarkies MN, Robinson S, Briffa T, Duffy SJ, Nelson M, Beltrame J, et al. Applying a framework to assess the impact of cardiovascular outcomes improvement research. Health Res Policy Syst. 2021;19(1):67. doi: 10.1186/s12961-021-00710-4. - DOI - PMC - PubMed
    1. Zaratin P, Battaglia MA, Abbracchio MP. Nonprofit foundations spur translational research. Trends Pharmacol Sci. 2014;35(11):552–555. doi: 10.1016/j.tips.2014.09.003. - DOI - PubMed
    1. Sarkies MN, Bowles KA, Skinner EH, Haas R, Lane H, Haines TP. The effectiveness of research implementation strategies for promoting evidence-informed policy and management decisions in healthcare: a systematic review. Implement Sci. 2017;12(1):1–20. doi: 10.1186/s13012-017-0662-0. - DOI - PMC - PubMed
    1. Sarkies MN, White J, Morris ME, Taylor NF, Williams C, O'Brien L, et al. Implementation of evidence-based weekend service recommendations for allied health managers: a cluster randomised controlled trial protocol. Implement Sci. 2018;13(1):1–14. doi: 10.1186/s13012-018-0752-7. - DOI - PMC - PubMed

Grants and funding

LinkOut - more resources