GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

Abstract

Objective: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

Study design and setting: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

Results: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

Conclusions: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

Keywords: Certainty of the evidence; GRADE; Nonrandomized studies; Quality of evidence; ROBINS; Risk of bias.

Figure 1. The process for using ROBINS-I

Figure 2. ROBINS-I risk of bias domains

In GRADE risk of bias is a domain and ROBINS-I domains are called items)

Figure 3. The current GRADE approach for certainty of evidence: initial certainty and rating domains

Figure 4. Assessing randomized trials and non-randomized studies with GRADE

Rating of risk of bias for randomized and non-randomized studies. Different tools are used to address for the two types of bodies of evidence (randomized and non-randomized). The risk of bias is then assessed across studies and integrated if possible. When ROBINS-I is used rating down by more than two levels is required for a body of evidence that is rated as critical on that instrument. *In practice this will generally lead to rating down by at least two levels to low or very low certainty for NRS. However, for results with large effects, or dose response, or results in which inference is strengthened by the plausible biases that exist, the extent of rating down may be lowered. We have not identified bodies of evidence in which a ROBINS-I assessment alone leads to no rating down, or rating down by only one level. How to integrate RCTs and NRS will be further discussed in upcoming GRADE guidance articles. ** For GRADE the corresponding terminology is not serious, serious, very serious and a fourth level of risk of bias. GRADE is currently exploring the appropriate term for the fourth level

Figure 5. GRADE approach for certainty of evidence with tools like ROBINS-I