Critical interpretation of Cochran's Q test depends on power and prior assumptions about heterogeneity

Abstract

We describe how an appropriate interpretation of the Q-test depends on its power to detect a given typical amount of between-study variance (τ(2)) as well as prior beliefs on heterogeneity. We illustrate these concepts in an evaluation of 1011 meta-analyses of clinical trials with ⩾4 studies and binary outcomes. These concepts can be seen as an application of the Bayes theorem. Across the 1011 meta-analyses, power to detect typical heterogeneity was low in most situations. Thus, usually a non-significant Q test did not change perceptibly prior convictions on heterogeneity. Conversely, significant results for the Q test typically augmented considerably the probability of heterogeneity. The posterior probability of heterogeneity depends on what τ(2) we want to detect. With the same approach, one may also estimate the posterior probability for the presence of heterogeneity that is large enough to annul statistically significant summary effects; that is half the average within-study variance of the combined studies; and that is able to change the summary effect estimate of the meta-analysis by 20%. The discussed analyses are exploratory, and may depend heavily on prior assumptions when power for the Q-test is low. Statistical heterogeneity in meta-analyses should be cautiously interpreted considering the power to detect a specific τ(2) and prior assumptions about the presence of heterogeneity. Copyright © 2010 John Wiley & Sons, Ltd.

Keywords: Bayes theorem; Q test; heterogeneity; meta‐analysis; power analysis.