Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Dec 21;7(1):242.
doi: 10.1186/s13643-018-0915-2.

The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures

Lisa Bero  1 , Nicholas Chartres  2 , Joanna Diong  3 , Alice Fabbri  2 , Davina Ghersi  4 , Juleen Lam  5   6 , Agnes Lau  7 , Sally McDonald  8 , Barbara Mintzes  9 , Patrice Sutton  5 , Jessica Louise Turton  8 , Tracey J Woodruff  5
Affiliations

The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures

Lisa Bero et al. Syst Rev. .

Abstract

Background: Systematic reviews, which assess the risk of bias in included studies, are increasingly used to develop environmental hazard assessments and public health guidelines. These research areas typically rely on evidence from human observational studies of exposures, yet there are currently no universally accepted standards for assessing risk of bias in such studies. The risk of bias in non-randomised studies of exposures (ROBINS-E) tool has been developed by building upon tools for risk of bias assessment of randomised trials, diagnostic test accuracy studies and observational studies of interventions. This paper reports our experience with the application of the ROBINS-E tool.

Methods: We applied ROBINS-E to 74 exposure studies (60 cohort studies, 14 case-control studies) in 3 areas: environmental risk, dietary exposure and drug harm. All investigators provided written feedback, and we documented verbal discussion of the tool. We inductively and iteratively classified the feedback into 7 themes based on commonalities and differences until all the feedback was accounted for in the themes. We present a description of each theme.

Results: We identified practical concerns with the premise that ROBINS-E is a structured comparison of the observational study being rated to the 'ideal' randomised controlled trial. ROBINS-E assesses 7 domains of bias, but relevant questions related to some critical sources of bias, such as exposure and funding source, are not assessed. ROBINS-E fails to discriminate between studies with a single risk of bias or multiple risks of bias. ROBINS-E is severely limited at determining whether confounders will bias study outcomes. The construct of co-exposures was difficult to distinguish from confounders. Applying ROBINS-E was time-consuming and confusing.

Conclusions: Our experience suggests that the ROBINS-E tool does not meet the need for an international standard for evaluating human observational studies for questions of harm relevant to public and environmental health. We propose that a simpler tool, based on empirical evidence of bias, would provide accurate measures of risk of bias and is more likely to meet the needs of the environmental and public health community.

Keywords: Cochrane; Environment; GRADE; Guidelines; Nutrition; Observational study; Public health guidelines; Quality assessment; Risk of bias; Systematic review.

PubMed Disclaimer

Conflict of interest statement

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

    1. National Research Council (U.S.) Board on Environmental Studies and Toxicology. Committee to Review the IRIS Process: Review of EPA’s integrated risk information system (IRIS) process. Washington, D.C: The National Academies Press; 2014. - PubMed
    1. Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect. 2014;122(7):711–718. doi: 10.1289/ehp.1307972. - DOI - PMC - PubMed
    1. United States. Congress. House. Committee on energy and commerce. Subcommittee on environment and the economy . H.R. ______, the TSCA modernization act of 2015: hearing before the subcommittee on environment and the economy of the committee on energy and commerce, house of representatives, one hundred fourteenth congress, first session, April 14, 2015. Washington, DC: Government publishing Office; 2015.
    1. 2016 NHMRC Standards for Guidelines [https://www.nhmrc.gov.au/guidelinesforguidelines/standards].
    1. Johnson PI, Koustas E, Vesterinen HM, Sutton P, Atchley DS, Kim AN, Campbell M, Donald JM, Sen S, Bero L, et al. Application of the navigation guide systematic review methodology to the evidence for developmental and reproductive toxicity of triclosan. Environ Int. 2016;92-93:716–728. doi: 10.1016/j.envint.2016.03.009. - DOI - PMC - PubMed

Publication types