ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Jonathan Ac Sterne¹, Miguel A Hernán², Barnaby C Reeves³, Jelena Savović⁴, Nancy D Berkman⁵, Meera Viswanathan⁶, David Henry⁷, Douglas G Altman⁸, Mohammed T Ansari⁹, Isabelle Boutron¹⁰, James R Carpenter¹¹, An-Wen Chan¹², Rachel Churchill¹³, Jonathan J Deeks¹⁴, Asbjørn Hróbjartsson¹⁵, Jamie Kirkham¹⁶, Peter Jüni¹⁷, Yoon K Loke¹⁸, Theresa D Pigott¹⁹, Craig R Ramsay²⁰, Deborah Regidor²¹, Hannah R Rothstein²², Lakhbir Sandhu²³, Pasqualina L Santaguida²⁴, Holger J Schünemann²⁵, Beverly Shea²⁶, Ian Shrier²⁷, Peter Tugwell²⁸, Lucy Turner²⁹, Jeffrey C Valentine³⁰, Hugh Waddington³¹, Elizabeth Waters³², George A Wells³³, Penny F Whiting³⁴, Julian Pt Higgins³⁵

Affiliations

¹ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK jonathan.sterne@bristol.ac.uk.
² Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; and Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; and Harvard-Massachusetts Institute of Technology Division of Health Sciences and Technology, Boston, Massachusetts, USA.
³ School of Clinical Sciences, University of Bristol, Bristol, BS2 8HW, UK.
⁴ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK.
⁵ Program on Health Care Quality and Outcomes, Division of Health Services and Social Policy Research, RTI International, Research Triangle Park, NC 27709, USA.
⁶ RTI-UNC Evidence-based Practice Center, RTI International, Research Triangle Park, NC 27709, USA.
⁷ Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada.
⁸ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK.
⁹ School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada.
¹⁰ METHODS Team, Centre of Epidemiology and Statistics Sorbonne Paris Cité Research, INSERM UMR 1153, University Paris Descartes, Paris, France.
¹¹ Department of Medical Statistics, London School of Hygiene and Tropical Medicine and MRC Clinical Trials Unit at UCL, London, UK.
¹² Women's College Research Institute, Department of Medicine, University of Toronto, Canada.
¹³ Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK.
¹⁴ Institute of Applied Health Research, University of Birmingham, Birmingham B15 2TT, UK.
¹⁵ Center for Evidence-Based Medicine, University of Southern Denmark & Odense University Hospital, 5000 Odense C, Denmark.
¹⁶ Department of Biostatistics, University of Liverpool, Liverpool, L69 3GL, UK.
¹⁷ Applied Health Research Centre (AHRC), Li Ka Shing Knowledge Institute of St Michael's Hospital, and Department of Medicine, University of Toronto, Toronto, Ontario, Canada.
¹⁸ Norwich Medical School, University of East Anglia, Norwich NR4 7TJ, UK.
¹⁹ School of Education, Loyola University Chicago, Chicago, IL 60611, USA.
²⁰ Health Services Research Unit, University of Aberdeen, Aberdeen, AB25 2ZD, UK.
²¹ Evidence Services, Kaiser Permanente, Care Management Institute, Oakland, CA 94612, USA.
²² Department of Management, Zicklin School of Business, Baruch College-CUNY, New York, NY 10010, USA.
²³ Division of General Surgery, University of Toronto, Toronto, Canada.
²⁴ Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, L8S 4K1, Canada.
²⁵ Departments of Clinical Epidemiology and Biostatistics and of Medicine, Cochrane Applicability and Recommendations Methods (GRADEing) Group, MacGRADE center, Ontario, L8N 4K1, Canada.
²⁶ Ottawa Hospital Research Institute, Center for Practice Changing Research and School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada.
²⁷ Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital, McGill University, Montreal, Quebec, Canada.
²⁸ Department of Medicine and School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, Ontario, Canada.
²⁹ Ottawa Hospital Research Institute, Ottawa, ON, Canada.
³⁰ University of Louisville, Louisville, KY 40292, USA.
³¹ International Initiative for Impact Evaluation, London School of Hygiene and Tropical Medicine, and London International Development Centre, London, UK.
³² Jack Brockhoff Child Health & Wellbeing Program, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, VIC 3010, Australia.
³³ School of Epidemiology, Public Health and Preventive Medicine and Director, Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Ontario, K1Y 4W7, Canada.
³⁴ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK; and National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK.
³⁵ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK.

PMID: 27733354
PMCID: PMC5062054
DOI: 10.1136/bmj.i4919

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Jonathan Ac Sterne et al. BMJ. 2016.

. 2016 Oct 12:355:i4919.

doi: 10.1136/bmj.i4919.

Authors

Affiliations

¹ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK jonathan.sterne@bristol.ac.uk.
² Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; and Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; and Harvard-Massachusetts Institute of Technology Division of Health Sciences and Technology, Boston, Massachusetts, USA.
³ School of Clinical Sciences, University of Bristol, Bristol, BS2 8HW, UK.
⁴ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK.
⁵ Program on Health Care Quality and Outcomes, Division of Health Services and Social Policy Research, RTI International, Research Triangle Park, NC 27709, USA.
⁶ RTI-UNC Evidence-based Practice Center, RTI International, Research Triangle Park, NC 27709, USA.
⁷ Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada.
⁸ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK.
⁹ School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada.
¹⁰ METHODS Team, Centre of Epidemiology and Statistics Sorbonne Paris Cité Research, INSERM UMR 1153, University Paris Descartes, Paris, France.
¹¹ Department of Medical Statistics, London School of Hygiene and Tropical Medicine and MRC Clinical Trials Unit at UCL, London, UK.
¹² Women's College Research Institute, Department of Medicine, University of Toronto, Canada.
¹³ Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK.
¹⁴ Institute of Applied Health Research, University of Birmingham, Birmingham B15 2TT, UK.
¹⁵ Center for Evidence-Based Medicine, University of Southern Denmark & Odense University Hospital, 5000 Odense C, Denmark.
¹⁶ Department of Biostatistics, University of Liverpool, Liverpool, L69 3GL, UK.
¹⁷ Applied Health Research Centre (AHRC), Li Ka Shing Knowledge Institute of St Michael's Hospital, and Department of Medicine, University of Toronto, Toronto, Ontario, Canada.
¹⁸ Norwich Medical School, University of East Anglia, Norwich NR4 7TJ, UK.
¹⁹ School of Education, Loyola University Chicago, Chicago, IL 60611, USA.
²⁰ Health Services Research Unit, University of Aberdeen, Aberdeen, AB25 2ZD, UK.
²¹ Evidence Services, Kaiser Permanente, Care Management Institute, Oakland, CA 94612, USA.
²² Department of Management, Zicklin School of Business, Baruch College-CUNY, New York, NY 10010, USA.
²³ Division of General Surgery, University of Toronto, Toronto, Canada.
²⁴ Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, L8S 4K1, Canada.
²⁵ Departments of Clinical Epidemiology and Biostatistics and of Medicine, Cochrane Applicability and Recommendations Methods (GRADEing) Group, MacGRADE center, Ontario, L8N 4K1, Canada.
²⁶ Ottawa Hospital Research Institute, Center for Practice Changing Research and School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada.
²⁷ Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital, McGill University, Montreal, Quebec, Canada.
²⁸ Department of Medicine and School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, Ontario, Canada.
²⁹ Ottawa Hospital Research Institute, Ottawa, ON, Canada.
³⁰ University of Louisville, Louisville, KY 40292, USA.
³¹ International Initiative for Impact Evaluation, London School of Hygiene and Tropical Medicine, and London International Development Centre, London, UK.
³² Jack Brockhoff Child Health & Wellbeing Program, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, VIC 3010, Australia.
³³ School of Epidemiology, Public Health and Preventive Medicine and Director, Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Ontario, K1Y 4W7, Canada.
³⁴ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK; and National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK.
³⁵ School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK.

PMID: 27733354
PMCID: PMC5062054
DOI: 10.1136/bmj.i4919

Abstract

Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation, but their results may be biased. It is therefore important to understand and appraise their strengths and weaknesses. We developed ROBINS-I (“Risk Of Bias In Non-randomised Studies - of Interventions”), a new tool for evaluating risk of bias in estimates of the comparative effectiveness (harm or benefit) of interventions from studies that did not use randomisation to allocate units (individuals or clusters of individuals) to comparison groups. The tool will be particularly useful to those undertaking systematic reviews that include non-randomised studies.

PubMed Disclaimer

Conflict of interest statement

All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: grants from Cochrane, MRC, and NIHR during the conduct of the study. Dr Carpenter reports personal fees from Pfizer, grants and non-financial support from GSK and grants from Novartis, outside the submitted work. Dr Reeves is a co-convenor of the Cochrane Non-Randomised Studies Methods Group. The authors report no other relationships or activities that could appear to have influenced the submitted work.

Figures

None — **Fig 1** Summary of the process of assessing risk of bias in a systematic review of non-randomised studies of interventions (NRSI)

See this image and copyright information in PMC

References

1. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312:1215-8. 10.1136/bmj.312.7040.1215 pmid:8634569. - DOI - PMC - PubMed
1. Feinstein AR. An additional basic science for clinical medicine: II. The limitations of randomized trials. Ann Intern Med 1983;99:544-50. 10.7326/0003-4819-99-4-544 pmid:6625387. - DOI - PubMed
1. Strom B. Overview of automated databases in pharmacoepidemiology. In: Strom BL, Hennessy S, eds. Pharmacoepidemiology. 5th ed. Wiley, 2012.
1. Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666-76. 10.1093/ije/dym018 pmid:17470488. - DOI - PubMed
1. Deeks JJ, Dinnes J, D’Amico R, et al. International Stroke Trial Collaborative Group European Carotid Surgery Trial Collaborative Group. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:iii-x, 1-173. 10.3310/hta7270 pmid:14499048. - DOI - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations
Medical
- ClinicalTrials.gov
- MedlinePlus Health Information

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Affiliations

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Medical