Publication bias and meta-analyses: a practical example

Abstract

Objectives: Publication bias is widely appreciated, but considerable time and effort are needed to locate and obtain data from unpublished randomized controlled trials (RCTs), those published in non-English language journals or those reported in the gray literature; for this publication, we will call this collection of trials the "gray+literature." However, excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions.

Methods: We aimed to explore and quantify the impact of inclusion of gray+ literature on the results of all completed individual patient data (IPD) reviews coordinated by our group (13 meta-analyses). For each IPD review, results were calculated for RCTs fully published in English language journals and RCTs fully published in English language journals and the gray+literature.

Results: The IPD meta-analyses based only on RCTs that were fully published in English language journals tended to give more favorable results than those that included RCTs from the gray+literature. Although in most cases the addition of gray+data gave less encouraging results, moving the estimated treatment effect toward a null result, the direction of effect was not always predictable.

Conclusions: We recommend that all systematic reviews should at least attempt to identify trials reported in the gray+literature and, where possible, obtain data from them.