The heterogeneity statistic I(2) can be biased in small meta-analyses

Abstract

Background: Estimated effects vary across studies, partly because of random sampling error and partly because of heterogeneity. In meta-analysis, the fraction of variance that is due to heterogeneity is estimated by the statistic I(2). We calculate the bias of I(2), focusing on the situation where the number of studies in the meta-analysis is small. Small meta-analyses are common; in the Cochrane Library, the median number of studies per meta-analysis is 7 or fewer.

Methods: We use Mathematica software to calculate the expectation and bias of I(2).

Results: I(2) has a substantial bias when the number of studies is small. The bias is positive when the true fraction of heterogeneity is small, but the bias is typically negative when the true fraction of heterogeneity is large. For example, with 7 studies and no true heterogeneity, I(2) will overestimate heterogeneity by an average of 12 percentage points, but with 7 studies and 80 percent true heterogeneity, I(2) can underestimate heterogeneity by an average of 28 percentage points. Biases of 12-28 percentage points are not trivial when one considers that, in the Cochrane Library, the median I(2) estimate is 21 percent.

Conclusions: The point estimate I(2) should be interpreted cautiously when a meta-analysis has few studies. In small meta-analyses, confidence intervals should supplement or replace the biased point estimate I(2).

Figure 1

The probability that a central chi-square variable Q is less than its degrees of freedom.

Figure 2

The expectation of I ² when ι ² = 0.

Figure 3

The expectation of I ² under a random-effects model, for ι ²=.1,.2, …,.9.

Figure 4

The expectation of I ² under a fixed-effects model, for ι ²=.1,.2, …,.9.