Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jun 26;2(7):e12094.
doi: 10.1002/cesm.12094. eCollection 2024 Jul.

Real-world evaluation of interconsensus agreement of risk of bias tools: A case study using risk of bias in nonrandomized studies-of interventions (ROBINS-I)

Samer Saadi  1   2 , Bashar Hasan  1   2 , Adel Kanaan  1 , Mohamed Abusalih  1   2 , Zin Tarakji  1   2 , Mustafa Sadek  1 , Ayla Shamsi Basha  1 , Mohammed Firwana  1   2 , Zhen Wang  1   2 , M Hassan Murad  1   2
Affiliations

Real-world evaluation of interconsensus agreement of risk of bias tools: A case study using risk of bias in nonrandomized studies-of interventions (ROBINS-I)

Samer Saadi et al. Cochrane Evid Synth Methods. .

Abstract

Background: Risk of bias (RoB) tools are critical in systematic reviews and affect subsequent decision-making. RoB tools should have adequate interrater reliability and interconsensus agreement. We present an approach of post hoc evaluation of RoB tools using duplicated studies that overlap systematic reviews.

Methods: Using a back-citation approach, we identified systematic reviews that used the Risk Of Bias In Nonrandomized Studies-of Interventions (ROBINS-I) tool and retrieved all the included primary studies. We selected studies that were appraised by more than one systematic review and calculated observed agreement and unweighted kappa comparing the different systematic reviews' assessments.

Results: We identified 903 systematic reviews that used the tool with 51,676 cited references, from which we eventually analyzed 171 duplicated studies assessed using ROBINS-I by different systematic reviewers. The observed agreement on ROBINS-I domains ranged from 54.9% (missing data domain) to 70.3% (deviations from intended interventions domain), and was 63.0% for overall RoB assessment of the study. Kappa coefficient ranged from 0.131 (measurement of outcome domain) to 0.396 (domains of confounding and deviations from intended interventions), and was 0.404 for overall RoB assessment of the study.

Conclusion: A post hoc evaluation of RoB tools is feasible by focusing on duplicated studies that overlap systematic review. ROBINS-I assessments demonstrated considerable variation in interconsensus agreement among various systematic reviewes that assessed the same study and outcome, suggesting the need for more intensive upfront work to calibrate systematic reviewers on how to identify context-specific information and agree on how to judge it.

Keywords: ROBINS‐I; interconsensus agreement; interrater reliability; methodological quality; risk of bias; systematic reviews.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
The process of study selection.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. Murad MH, Montori VM, Ioannidis JPA, et al. How to read a systematic review and meta‐analysis and apply the results to patient care: users' guides to the medical literature. JAMA. 2014;312(2):171‐179. 10.1001/jama.2014.5559 - DOI - PubMed
    1. Lo CKL, Mertz D, Loeb M. Newcastle‐Ottawa Scale: comparing reviewers' to authors' assessments. BMC Med Res Methodol. 2014;14:45. 10.1186/1471-2288-14-45 - DOI - PMC - PubMed
    1. Zhang Y, Huang L, Wang D, Ren P, Hong Q, Kang D. The ROBINS‐I and the NOS had similar reliability but differed in applicability: a random sampling observational studies of systematic reviews/meta‐analysis. J Evid Based Med. 2021;14(2):112‐122. 10.1111/jebm.12427 - DOI - PubMed
    1. Hartling L, Hamm M, Milne A, et al. Validity and Inter‐Rater Reliability Testing of Quality Assessment Instruments. Agency for Healthcare Research and Quality; 2012. - PubMed
    1. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS‐I: a tool for assessing risk of bias in non‐randomised studies of interventions. BMJ. 2016;355:i4919. 10.1136/bmj.i4919 - DOI - PMC - PubMed

LinkOut - more resources