The challenges of measuring social cohesion in public health research: A systematic review and ecometric meta-analysis

Abstract

The relationship between social cohesion and health has been studied for decades. Yet, due to the contextual nature of this concept, measuring social cohesion remains challenging. Using a meta-analytical framework, this review's goal was to study the ecometric measurement properties of social cohesion in order to describe dissimilarities in its measurement as well as bring a new perspective on the empirical usefulness of the concept itself. To this end, we analysed if, and to what extent, contextual-level reliability and intersubjective agreement of 78 social cohesion measurements varied under different measurement conditions like measurement instrument, spatial unit, ecometric model specification, or region. We found consistent evidence for the contextual nature of social cohesion, however, most variation existed between individuals, not contexts. While contextual dependence in response behaviour was fairly insensitive to item choices, population size within chosen spatial units of social cohesion measurements mattered. Somewhat counterintuitively, using spatial units with, on average, fewer residents did not yield systematically superior ecometric properties. Instead, our results underline that precise theory about the relevant contextual units of causal relationships between social cohesion and health is vital and cannot be replaced by empirical analysis. Although adjustment for respondent's characteristics had only small effects on ecometric properties, potential pitfalls of this analytic strategy are discussed in this paper. Finally, acknowledging the sensitivity of measuring social cohesion, we derived recommendations for future studies investigating the effects of contextual-level social characteristics on health.

Keywords: Contextual characteristics; Contextual effects; Ecometrics; Intracluster correlation coefficient; Meta-analysis; Multilevel modelling; Social cohesion; Social environment.

Fig. 1

A scenario in which the ICC would be 1. Within each context, all respondents agree on the level of a contextual characteristic (values within each box), thus, the within-variance equals 0. However, contexts vary in their level of this characteristic (values above each box). In the case illustrated above, the between-variance equals 0.3125. Following equation (4) the ICC equals 0.3125/(0.3125 + 0) = 1.

Fig. 2

A scenario in which the ICC would be 0. Respondents answer differently within contexts (values next to individuals) but the level of the contextual characteristic (value above each box) does not vary across contexts. Thus, item responses of individuals are independent of the chosen contextual level and there is no variation between contexts, only within contexts.

Fig. 3

The between-context variation is also dependent on the chosen contextual level. Redefining the contextual level chosen in Fig. 2 (black dashed lines instead of grey lines) yields an entirely different ICC compared to the ICC in the scenario illustrated in Fig. 2.

Fig. 4

Study selection process.

Fig. 5

Internal consistencies of social cohesion instruments measured by Cronbach's alpha (alpha) and contextual-level reliabilities (lambda) of social cohesion instruments. Modified versions include measurements that did not use the same items and/or response scales but only differed marginally compared to the original measurement instrument. See supplementary material for explanations of item categories. Contextual-level reliabilities were obtained by slightly different methods – see Fig. S1 or Table S1 for method-specific reliabilities.

Fig. 6

An overview of observations for which the ICC was available.

Fig. 7

Distribution of estimated inter-subjective agreement (ICC) across different measurement instruments of social cohesion.

Fig. 8

Inter-subjective agreement (ICC) against average number of residents within chosen spatial units by comparable measurements of social cohesion. The scale of the x-axis is logarithmic and the y-axis only ranges from 0 to 0.4 as there were no ICCs greater than 0.4 in our data.