The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey

Abstract

Background: Statistical tests for funnel-plot asymmetry are common in meta-analyses. Inappropriate application can generate misleading inferences about publication bias. We aimed to measure, in a survey of meta-analyses, how frequently the application of these tests would be not meaningful or inappropriate.

Methods: We evaluated all meta-analyses of binary outcomes with é 3 studies in the Cochrane Database of Systematic Reviews (2003, issue 2). A separate, restricted analysis was confined to the largest meta-analysis in each of the review articles. In each meta-analysis, we assessed whether criteria to apply asymmetry tests were met: no significant heterogeneity, I2 < 50%, é 10 studies (with statistically significant results in at least 1) and ratio of the maximal to minimal variance across studies > 4. We performed a correlation and 2 regression asymmetry tests and evaluated their concordance. Finally, we sampled 60 meta-analyses from print journals in 2005 that cited use of the standard regression test.

Results: A total of 366 of 6873 (5%) and 98 of 846 meta-analyses (12%) in the wider and restricted Cochrane data set, respectively, would have qualified for use of asymmetry tests. Asymmetry test results were significant in 7%-18% of the meta-analyses. Concordance between the 3 tests was modest (estimated k 0.33-0.66). Of the 60 journal meta-analyses, 7 (12%) would qualify for asymmetry tests; all 11 claims for identification of publication bias were made in the face of large and significant heterogeneity.

Interpretation: Statistical conditions for employing asymmetry tests for publication bias are absent from most meta-analyses; yet, in medical journals these tests are performed often and interpreted erroneously.

Fig. 1: Venn diagrams showing the overlap of the subsets of meta-analyses according to our chosen criteria (diagrams to the left: ≥ 1 study with statistically significant results; ≥ 10 studies in the meta-analysis; I² < 50% with nonsignificant Q; ratio of extreme study variances > 4). For comparison, results when a set of very lenient criteria (right: ≥ 1 significant study; ≥ 5 studies; I² < 50% regardless of Q; extreme study variances > 2) is used are also depicted. Each set of criteria is likewise shown for our wider data set of meta-analyses (upper diagrams: n = 6873) and for the restricted data set of 1 meta-analysis per systematic review (lower diagrams: n = 846). Shading indicates categories in which substantially more studies met criteria.

Fig. 2: Venn diagrams disclosing modest concordance in the application of the 3 funnel-plot asymmetry tests to statistically significant results in the wider data set of 6873 meta-analyses (left) and in the restricted data set of 846 meta-analyses (right). Data inside the circles refer only to meta-analyses with significant results with the corresponding test (p < 0.10).