Heterogeneity in effect size estimates

Abstract

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.

Keywords: generalizability; heterogeneity; metascience.

Fig. 1.

The figure illustrates the effective PSP_G, i.e., the ratio of true positive results to the total number of positive classifications in the presence of heterogeneity, for different prior probabilities for the alternative hypothesis being genuinely true (ϕ), as a function of the heterogeneity factor H for a two-tailed z-test with nominal statistical power of π = 90% at the nominal α = 5% level.

Fig. 2.

Empirical estimates of population, design, and analytical heterogeneity. (A) The figure shows estimates of the heterogeneity factor H for 70 estimates from 13 papers isolating population heterogeneity (–66), 11 estimates from two papers isolating design heterogeneity (67, 68), and five estimates from three papers isolating analytical heterogeneity (–71). The vertical reference lines indicate benchmark levels for small, medium, and large heterogeneity based on I² values of 25% (H = 1.15), 50% (H = 1.41), and 75% (H = 2), respectively. (B) The figure shows box plots of the distribution of heterogeneity factors H, separated by the source of heterogeneity, illustrated in panel (A).