Misunderstandings about Q and 'Cochran's Q test' in meta-analysis

Abstract

Many meta-analyses report using 'Cochran's Q test' to assess heterogeneity of effect-size estimates from the individual studies. Some authors cite work by W. G. Cochran, without realizing that Cochran deliberately did not use Q itself to test for heterogeneity. Further, when heterogeneity is absent, the actual null distribution of Q is not the chi-squared distribution assumed for 'Cochran's Q test'. This paper reviews work by Cochran related to Q. It then discusses derivations of the asymptotic approximation for the null distribution of Q, as well as work that has derived finite-sample moments and corresponding approximations for the cases of specific measures of effect size. Those results complicate implementation and interpretation of the popular heterogeneity index I(2) . Also, it turns out that the test-based confidence intervals used with I(2) are based on a fallacious approach. Software that outputs Q and I(2) should use the appropriate reference value of Q for the particular measure of effect size and the current meta-analysis. Q is a key element of the popular DerSimonian-Laird procedure for random-effects meta-analysis, but the assumptions of that procedure and related procedures do not reflect the actual behavior of Q and may introduce bias. The DerSimonian-Laird procedure should be regarded as unreliable.

Keywords: DerSimonian-Laird procedure; I2; heterogeneity; random-effects meta-analysis.