Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review

Abstract

Background: Organizational culture, organizational climate, and implementation climate are key organizational constructs that influence the implementation of evidence-based practices. However, there has been little systematic investigation of the availability of psychometrically strong measures that can be used to assess these constructs in behavioral health. This systematic review identified and assessed the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs as defined by the Consolidated Framework for Implementation Research (CFIR) and Ehrhart and colleagues.

Methods: Data collection involved search string generation, title and abstract screening, full-text review, construct assignment, and citation searches for all known empirical uses. Data relevant to nine psychometric criteria from the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) were extracted: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. Extracted data for each criterion were rated on a scale from -1 ("poor") to 4 ("excellent"), and each measure was assigned a total score (highest possible score = 36) that formed the basis for head-to-head comparisons of measures for each focal construct.

Results: We identified full measures or relevant subscales of broader measures for organizational culture (n = 21), organizational climate (n = 36), implementation climate (n = 2), tension for change (n = 2), compatibility (n = 6), relative priority (n = 2), organizational incentives and rewards (n = 3), goals and feedback (n = 3), and learning climate (n = 2). Psychometric evidence was most frequently available for internal consistency and norms. Information about other psychometric properties was less available. Median ratings for psychometric properties across categories of measures ranged from "poor" to "good." There was limited evidence of responsiveness or predictive validity.

Conclusion: While several promising measures were identified, the overall state of measurement related to these constructs is poor. To enhance understanding of how these constructs influence implementation research and practice, measures that are sensitive to change and predictive of key implementation and clinical outcomes are required. There is a need for further testing of the most promising measures, and ample opportunity to develop additional psychometrically strong measures of these important constructs.

Plain language summary: Organizational culture, organizational climate, and implementation climate can play a critical role in facilitating or impeding the successful implementation and sustainment of evidence-based practices. Advancing our understanding of how these contextual factors independently or collectively influence implementation and clinical outcomes requires measures that are reliable and valid. Previous systematic reviews identified measures of organizational factors that influence implementation, but none focused explicitly on behavioral health; focused solely on organizational culture, organizational climate, and implementation climate; or assessed the evidence base of all known uses of a measure within a given area, such as behavioral health-focused implementation efforts. The purpose of this study was to identify and assess the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs that have been used in behavioral health-focused implementation research. We identified 21 measures of organizational culture, 36 measures of organizational climate, 2 measures of implementation climate, 2 measures of tension for change, 6 measures of compatibility, 2 measures of relative priority, 3 measures of organizational incentives and rewards, 3 measures of goals and feedback, and 2 measures of learning climate. Some promising measures were identified; however, the overall state of measurement across these constructs is poor. This review highlights specific areas for improvement and suggests the need to rigorously evaluate existing measures and develop new measures.

Keywords: Implementation science; behavioral health; implementation climate; measurement; mental health; organizational climate; organizational culture; psychometric properties; reliability; validity.

Figure 1.

Head-to-head comparison of measures of organizational culture.

Figure 2.

Head-to-head comparison of measures of organizational climate.

Figure 3.

Head-to-head comparison of measures of implementation climate.

Figure 4.

Head-to-head comparison of measures of tension for change.

Figure 5.

Head-to-head comparison of measures of compatibility.

Figure 6.

Head-to-head comparison of measures of relative priority.

Figure 7.

Head-to-head comparison of measures of organizational incentives and rewards.

Figure 8.

Head-to-head comparison of measures of goals and feedback.

Figure 9.

Head-to-head comparison of measures of learning climate.

Figure 10.

PRISMA diagram for organizational culture. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 11.

PRISMA diagram for implementation climate. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 12.

PRISMA diagram for tension for change. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 13.

PRISMA diagram for compatibility. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 14.

PRISMA diagram for relative priority. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 15.

PRISMA diagram for organizational incentives and rewards. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 16.

PRISMA diagram for goals and feedback. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.

Figure 17.

PRISMA diagram for learning climate. Note that the number of articles identified did not equal the number of measures included in the analysis because in some cases a single measure was identified in multiple articles, and in others, multiple measures were identified in a single article.