Search for unpublished data by systematic reviewers: an audit

Abstract

Objectives: We audited a selection of systematic reviews published in 2013 and reported on the proportion of reviews that researched for unpublished data, included unpublished data in analysis and assessed for publication bias.

Design: Audit of systematic reviews.

Data sources: We searched PubMed and Ovid MEDLINE In-Process & Other Non-Indexed Citations between 1 January 2013 and 31 December 2013 for the following journals: Journal of the American Medical Association, The British Medical Journal, Lancet, Annals of Internal Medicine and the Cochrane Database of Systematic Reviews. We also searched the Cochrane Library and included 100 randomly selected Cochrane reviews.

Eligibility criteria: Systematic reviews published in 2013 in the selected journals were included. Methodological reviews were excluded.

Data extraction and synthesis: Two reviewers independently reviewed each included systematic review. The following data were extracted: whether the review searched for grey literature or unpublished data, the sources searched, whether unpublished data were included in analysis, whether publication bias was assessed and whether there was evidence of publication bias.

Main findings: 203 reviews were included for analysis. 36% (73/203) of studies did not describe any attempt to obtain unpublished studies or to search grey literature. 89% (116/130) of studies that sought unpublished data found them. 33% (68/203) of studies included an assessment of publication bias, and 40% (27/68) of these found evidence of publication bias.

Conclusion: A significant fraction of systematic reviews included in our study did not search for unpublished data. Publication bias may be present in almost half the published systematic reviews that assessed for it. Exclusion of unpublished data may lead to biased estimates of efficacy or safety in systematic reviews.

Keywords: Review; Systematic Publication Bias Unpublished Data.