Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2017 Apr 12;7(4):e014772.
doi: 10.1136/bmjopen-2016-014772.

Systematic approach to evaluating and confirming the utility of a suite of national health system performance (HSP) indicators in Canada: a modified Delphi study

Omid Fekri  1 , Kira Leeb  1 , Yana Gurevich  1
Affiliations

Systematic approach to evaluating and confirming the utility of a suite of national health system performance (HSP) indicators in Canada: a modified Delphi study

Omid Fekri et al. BMJ Open. .

Abstract

Objectives: Evaluating an existing suite of health system performance (HSP) indicators for continued reporting using a systematic criteria-based assessment and national consensus conference.

Design: Modified Delphi approach with technical and leadership groups, an online survey of stakeholders and convening a national consensus conference.

Setting: A national health information steward, the Canadian Institute for Health Information (CIHI).

Participants: A total of 73 participants, comprised 61 conference attendants/stakeholders from across Canada and 12 national health information steward staff.

Primary and secondary outcome measures: Indicator dispositions of retention, additional stakeholder consultation, further redevelopment or retirement.

Results: 4 dimensions (usability, importance, scientific soundness and feasibility) typically used to select measures for reporting were expanded to 18 criteria grouped under the 4 dimensions through a process of research and testing. Definitions for each criterion were developed and piloted. Once the definitions were established, 56 of CIHI's publicly reported HSP indicators were evaluated against the criteria using modified Delphi approaches. Of the 56 HSP indicators evaluated, 9 measures were ratified for retirement, 7 were identified for additional consultation and 3 for further research and development. A pre-Consensus Conference survey soliciting feedback from stakeholders on indicator recommendations received 48 responses (response rate of 79%).

Conclusions: A systematic evaluation of HSP indicators informed the development of objective recommendations for continued reporting. The evaluation was a fruitful exercise to identify technical considerations for calculating indicators, furthering our understanding of how measures are used by stakeholders, as well as harmonising actions that could be taken to ensure relevancy, reduce indicator chaos and build consensus with stakeholders.

Keywords: Consensus methods; Evaluation; Health indicators; Health system performance.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Similar articles

See all similar articles

Cited by

References

    1. Klazinga N, Fischer C, ten Asbroek A. Health services research related to performance indicators and benchmarking in Europe. J Health Serv Res Policy 2011:16:38–47. http://hsr.sagepub.com/content/16/suppl_2/38.long (accessed 16 Oct 2016). 10.1258/jhsrp.2011.011042 - DOI - PubMed
    1. Kelley E, Arispe I, Homes J. Beyond the initial indicators: lessons from the OECD Health Care Quality Indicators Project and the US National Healthcare Quality Report. Int J Qual Health Care 2006;18(Suppl 1):45–51. http://intqhc.oxfordjournals.org/content/18/suppl_1/45.long (accessed 16 Oct 2016). 10.1093/intqhc/mzl027 - DOI - PubMed
    1. Carinci F, Van Gool K, Mainz J, et al. . Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care 2015;27:137–46. http://intqhc.oxfordjournals.org/content/27/2/137 (accessed 16 Oct 2016). - PubMed
    1. Canadian Institute for Health Information. Roadmap initiative … launching the process. Ottawa: 2000. https://www.cihi.ca/en/profile_roadmap_launch_pdf_en.pdf (accessed 16 Oct 2016).
    1. Canadian Institute for Health Information. A performance measurement framework for the Canadian health system. Ottawa: 2013. https://secure.cihi.ca/free_products/HSP-Framework-ENweb.pdf (accessed 16 Oct 2016).

Publication types

LinkOut - more resources