Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Mar 14;8(3):e019703.
doi: 10.1136/bmjopen-2017-019703.

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review

Matthew J Page  1   2 , Joanne E McKenzie  1 , Julian P T Higgins  2
Affiliations

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review

Matthew J Page et al. BMJ Open. .

Abstract

Background: Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.

Methods: We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.

Results: We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at 'high' risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as 'high' risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.

Conclusion: There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.

Keywords: bias (epidemiology); checklist; publication bias; review literature as topic.

PubMed Disclaimer

Conflict of interest statement

Competing interests: JPTH led or participated in the development of four of the included tools (the current Cochrane risk of bias tool for randomised trials, the RoB 2.0 tool for assessing risk of bias in randomised trials, the ROBINS-I tool for assessing risk of bias in non-randomised studies of interventions and the framework for assessing quality of evidence from a network meta-analysis). MJP participated in the development of one of the included tools (the RoB 2.0 tool for assessing risk of bias in randomised trials). All authors are participating in the development of a new tool for assessing risk of reporting biases in systematic reviews.

Figures

Figure 1
Figure 1
Flow diagram of identification, screening and inclusion of studies. aRecords identified from Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar. bRecords identified from screening references of included articles. SR, systematic review.
See this image and copyright information in PMC

References

    1. Chan AW, Song F, Vickers A, et al. . Increasing value and reducing waste: addressing inaccessible research. Lancet 2014;383:257–66. 10.1016/S0140-6736(13)62296-5 - DOI - PMC - PubMed
    1. Song F, Parekh S, Hooper L, et al. . Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:8 10.3310/hta14080 - DOI - PubMed
    1. Kirkham JJ, Dwan KM, Altman DG, et al. . The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365 10.1136/bmj.c365 - DOI - PubMed
    1. Sterne JA, Hernán MA, Reeves BC, et al. . ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919 10.1136/bmj.i4919 - DOI - PMC - PubMed
    1. Schmucker C, Schell LK, Portalupi S, et al. . Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One 2014;9:e114023 10.1371/journal.pone.0114023 - DOI - PMC - PubMed

Publication types

LinkOut - more resources