Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2021 Oct 25;10(1):273.
doi: 10.1186/s13643-021-01819-x.

A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2

R Perry  1 , A Whitmarsh  2 , V Leach  3 , P Davies  3   4
Affiliations
Review

A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2

R Perry et al. Syst Rev. .

Abstract

Background: AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions.

Methods: Three reviewers independently assessed 31 systematic reviews using both tools. The inter-rater reliability of all sub-sections using each instrument (AMSTAR-2 and ROBIS) was calculated using Gwet's agreement coefficient (AC1 for unweighted analysis and AC2 for weighted analysis).

Results: Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher.

Conclusion: ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.

Keywords: AMSTAR-2; Methodological quality; ROBIS; Risk of bias; Systematic reviews.

PubMed Disclaimer

Conflict of interest statement

Rachel Perry was an author on two of the papers under review. Philippa Davies was involved in the development of ROBIS. The other authors declare that they have no competing interests.

Figures

Fig. 1
Fig. 1
Gwet’s statistic for the inter-rater agreement for AMSTAR-2 questions
Fig. 2
Fig. 2
Gwet’s statistic for the inter-rater agreement for ROBIS questions and domains
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Murad H, Asi N, Alsawas M. New evidence pyramid. BMJ Evid Based Med. 2016;21:125–127. - PMC - PubMed
    1. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;15(7):10. - PMC - PubMed
    1. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62:1013–1020. - PubMed
    1. Burda BU, Holmer HK, Norris SL. Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement. Syst Rev. 2016;5:58. - PMC - PubMed
    1. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. - PMC - PubMed

Publication types

LinkOut - more resources