Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Jul:111:105-114.
doi: 10.1016/j.jclinepi.2018.01.012. Epub 2018 Feb 9.

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

Holger J Schünemann  1 , Carlos Cuello  2 , Elie A Akl  3 , Reem A Mustafa  4 , Jörg J Meerpohl  5 , Kris Thayer  6 , Rebecca L Morgan  2 , Gerald Gartlehner  7 , Regina Kunz  8 , S Vittal Katikireddi  9 , Jonathan Sterne  10 , Julian Pt Higgins  10 , Gordon Guyatt  11 ; GRADE Working Group
Affiliations

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

Holger J Schünemann et al. J Clin Epidemiol. 2019 Jul.

Abstract

Objective: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

Study design and setting: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

Results: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

Conclusions: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

Keywords: Certainty of the evidence; GRADE; Nonrandomized studies; Quality of evidence; ROBINS; Risk of bias.

PubMed Disclaimer

Conflict of interest statement

Conflict of interest

HJS has no direct financial conflict of interest and other authors have not declared financial conflicts of interest. Part of the work has been presented scientific conferences and at GRADE Working Group meetings. This article has been officially endorsed by the GRADE Working Group.

Figures

Figure 1
Figure 1. The process for using ROBINS-I
Figure 2
Figure 2. ROBINS-I risk of bias domains
In GRADE risk of bias is a domain and ROBINS-I domains are called items)
Figure 3
Figure 3. The current GRADE approach for certainty of evidence: initial certainty and rating domains
Figure 4
Figure 4. Assessing randomized trials and non-randomized studies with GRADE
Rating of risk of bias for randomized and non-randomized studies. Different tools are used to address for the two types of bodies of evidence (randomized and non-randomized). The risk of bias is then assessed across studies and integrated if possible. When ROBINS-I is used rating down by more than two levels is required for a body of evidence that is rated as critical on that instrument. *In practice this will generally lead to rating down by at least two levels to low or very low certainty for NRS. However, for results with large effects, or dose response, or results in which inference is strengthened by the plausible biases that exist, the extent of rating down may be lowered. We have not identified bodies of evidence in which a ROBINS-I assessment alone leads to no rating down, or rating down by only one level. How to integrate RCTs and NRS will be further discussed in upcoming GRADE guidance articles. ** For GRADE the corresponding terminology is not serious, serious, very serious and a fourth level of risk of bias. GRADE is currently exploring the appropriate term for the fourth level
Figure 5
Figure 5. GRADE approach for certainty of evidence with tools like ROBINS-I
See this image and copyright information in PMC

References

    1. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. - PubMed
    1. Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knotterus A. GRADE guidelines: A new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2010 - PubMed
    1. Puhan MA, Schunemann HJ, Murad MH, Li T, Brignardello-Petersen R, Singh JA, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630. - PubMed
    1. Schunemann HJ, Best D, Vist G, Oxman AD, Group GW Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. CMAJ. 2003;169(7):677–80. - PMC - PubMed
    1. Schunemann HJ, Oxman AD, Brozek J, Glasziou P, Bossuyt P, Chang S, et al. GRADE: assessing the quality of evidence for diagnostic recommendations. ACP J Club. 2008;149(6):2. - PubMed