Direction and impact of language bias in meta-analyses of controlled trials: empirical study

Abstract

Background: Excluding clinical trials reported in languages other than English from meta-analyses may introduce bias and reduce the precision of combined estimates of treatment effects. We examined the influence of trials published in languages other than English on combined estimates and conclusions of published meta-analyses.

Methods: We searched journals and the Cochrane Database of Systematic Reviews for meta-analyses of at least five trials with binary outcomes that were based on comprehensive literature searches without language restrictions. We compared estimates of treatment effects from trials published in languages other than English to those from trials published in English, and assessed the impact of restricting meta-analyses to trials published in English.

Results: We identified 303 meta-analyses: 159 (52.4%) employed comprehensive literature searches of which 50 included 485 English and 115 non-English language trials. Non-English language trials included fewer participants (median 88 versus 116, P = 0.006) and were more likely to produce significant results at P < 0.05 (41.7% versus 31.3%, P = 0.033). The methodological quality of non-English language trials tended to be lower than that of trials published in English. Estimates of treatment effects were on average 16% (95% CI : 3-26%) more beneficial in non-English-language trials than in English-language trials. In 29 (58.0%) meta-analyses the change in effect estimates after exclusion of non-English language trials was less than 5%. In the remaining meta-analyses, 5 (10.0%) showed more benefit and 16 (32.0%) less benefit after exclusion of non-English language trials.

Conclusions: This retrospective analysis suggests that excluding trials published in languages other than English has generally little effect on summary treatment effect estimates. The importance of non-English language trials is, however, difficult to predict for individual systematic reviews. Comprehensive literature searches followed by a careful assessment of trial quality are required to assess the contribution of all relevant trials, independent of language of publication.