Comparisons of various estimates of the I2 statistic for quantifying between-study heterogeneity in meta-analysis

Abstract

Assessing heterogeneity between studies is a critical step in determining whether studies can be combined and whether the synthesized results are reliable. The $I^2$ statistic has been a popular measure for quantifying heterogeneity, but its usage has been challenged from various perspectives in recent years. In particular, it should not be considered an absolute measure of heterogeneity, and it could be subject to large uncertainties. As such, when using $I^2$ to interpret the extent of heterogeneity, it is essential to account for its interval estimate. Various point and interval estimators exist for $I^2$. This article summarizes these estimators. In addition, we performed a simulation study under different scenarios to investigate preferable point and interval estimates of $I^2$. We found that the Sidik-Jonkman method gave precise point estimates for $I^2$ when the between-study variance was large, while in other cases, the DerSimonian-Laird method was suggested to estimate $I^2$. When the effect measure was the mean difference or the standardized mean difference, the $Q$-profile method, the Biggerstaff-Jackson method, or the Jackson method was suggested to calculate the interval estimate for $I^2$ due to reasonable interval length and more reliable coverage probabilities than various alternatives. For the same reason, the Kulinskaya-Dollinger method was recommended to calculate the interval estimate for $I^2$ when the effect measure was the log odds ratio.

Keywords: Confidence interval; I2 statistic; coverage probability; heterogeneity; meta-analysis.