Impact of risk of generalizability biases in adult obesity interventions: A meta-epidemiological review and meta-analysis

Abstract

Biases introduced in early-stage studies can lead to inflated early discoveries. The risk of generalizability biases (RGBs) identifies key features of feasibility studies that, when present, lead to reduced impact in a larger trial. This meta-study examined the influence of RGBs in adult obesity interventions. Behavioral interventions with a published feasibility study and a larger scale trial of the same intervention (e.g., pairs) were identified. Each pair was coded for the presence of RGBs. Quantitative outcomes were extracted. Multilevel meta-regression models were used to examine the impact of RGBs on the difference in the effect size (ES, standardized mean difference) from pilot to larger scale trial. A total of 114 pairs, representing 230 studies, were identified. Overall, 75% of the pairs had at least one RGB present. The four most prevalent RGBs were duration (33%), delivery agent (30%), implementation support (23%), and target audience (22%) bias. The largest reductions in the ES were observed in pairs where an RGB was present in the pilot and removed in the larger scale trial (average reduction ES -0.41, range -1.06 to 0.01), compared with pairs without an RGB (average reduction ES -0.15, range -0.18 to -0.14). Eliminating RGBs during early-stage testing may result in improved evidence.

Keywords: intervention; pilot; scaling; translation.

FIGURE 1

Diagram of search procedures of locating pilot studies and larger scale trial pairs

FIGURE 2

PRISMA diagram of systematic literature search and final studies included in analyses

FIGURE 3

Classification of the presence (red circle) and absence (green circle) risk of generalizability biases across pilot and larger scale trial pairs

FIGURE 4

Forest plot of the change in the standardized difference in means (SDM) of the presence, absence, or carry forward of risk of generalizability biases from a pilot/feasibility study to a larger scale trial. No pairs contained directional conclusion bias in both the pilot and larger scale trial. Intervention duration, intervention intensity, and measurement describe differences between smaller and larger scale studies, so they cannot be present in both studies

FIGURE 5

Z value distribution of outcomes in larger scale trials by the absence (green line) of risk of generalizability bias (RGB), RGB present in pilot and absent in larger scale trial (red distribution), and RGB present in both pilot and larger scale trial (blue distribution). Positive z values indicate that the intervention was better than the control group, and negative z values indicate that the control group was better than the intervention group. Solid vertical lines represent z ± 1.96 (p = 0.05); dashed vertical lines represent z ± 2.58 (p = 0.005)