Quantifying heterogeneity in individual participant data meta-analysis with binary outcomes

Abstract

Background: In meta-analyses (MA), effect estimates that are pooled together will often be heterogeneous. Determining how substantial heterogeneity is is an important aspect of MA.

Method: We consider how best to quantify heterogeneity in the context of individual participant data meta-analysis (IPD-MA) of binary data. Both two- and one-stage approaches are evaluated via simulation study. We consider conventional I ² and R ² statistics estimated via a two-stage approach and R ² estimated via a one-stage approach. We propose a simulation-based intraclass correlation coefficient (ICC) adapted from Goldstein et al. to estimate the I ², from the one-stage approach.

Results: Results show that when there is no effect modification, the estimated I ² from the two-stage model is underestimated, while in the one-stage model, it is overestimated. In the presence of effect modification, the estimated I ² from the one-stage model has better performance than that from the two-stage model when the prevalence of the outcome is high. The I ² from the two-stage model is less sensitive to the strength of effect modification when the number of studies is large and prevalence is low.

Conclusions: The simulation-based I ² based on a one-stage approach has better performance than the conventional I ² based on a two-stage approach when there is strong effect modification with high prevalence.

Keywords: Heterogeneity; I 2; Individual participant data meta-analysis (IPD-MA); Two-stage and one-stage approaches.

Fig. 1

True τ ² versus estimated τ ². The estimated between-study variances from a conventional two-stage model and a simulation-based one-stage mode are compared with the true between-study variance

Fig. 2

True τ ² versus estimated I ². The estimated I ² from a conventional two-stage model and a simulation-based one-stage model are compared with the true between-study variance. The dashed line and dotted line represented the estimated I ² from the two-stage and one-stage models based on its median value across 1000 datasets