Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies

Abstract

Objective: To compare A Measurement Tool to Assess Systematic Reviews (AMSTAR 2) with a tool to assess risk of bias in systematic reviews (ROBIS) in terms of validity, reliability, and applicability.

Study design and setting: We analyzed 30 systematic reviews (SRs) that included randomized and nonrandomized studies, with Cochrane and non-Cochrane SRs sampled in 1:1 ratio. Four reviewers assessed independently all 30 SRs with AMSTAR 2, followed by ROBIS. We calculated Fleiss' Kappa as a measure of inter-rater reliability (IRR) across 4 raters.

Results: The IRR for scoring the overall confidence in the SRs with AMSTAR 2 and the overall domain in ROBIS was fair (AMSTAR 2: κ = 0.30, 95% [confidence interval] CI: 0.17 to 0.43; ROBIS: κ = 0.28, 95% CI: 0.13 to 0.42). AMSTAR 2 confidence in review ratings strongly correlated with the overall domain rating in ROBIS (Spearman r_s = 0.84). Mean time for scoring AMSTAR 2 was slightly higher than for ROBIS (18 vs. 16 min), with huge differences between the reviewers.

Conclusion: Both AMSTAR 2 and ROBIS can be applied to SRs including both randomized controlled trials (RCTs) and non-RCTs. Measurement properties of ROBIS seemed not to be much different when comparing with other studies that include only SRs of RCTs.

Keywords: AMSTAR; AMSTAR 2; Methodological quality; ROBIS; Risk of bias; Systematic reviews.