Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Meta-Analysis
. 2011 Sep 7;2011(9):MR000025.
doi: 10.1002/14651858.MR000025.pub2.

Blinded versus unblinded assessments of risk of bias in studies included in a systematic review

Kate Morissette  1 , Andrea C Tricco, Tanya Horsley, Maggie H Chen, David Moher
Affiliations
Meta-Analysis

Blinded versus unblinded assessments of risk of bias in studies included in a systematic review

Kate Morissette et al. Cochrane Database Syst Rev. .

Abstract

Background: The importance of appraising the risk of bias of studies included in systematic reviews is well-established. However, uncertainty remains surrounding the method by which risk of bias assessments should be conducted. Specifically, no summary of evidence exists as to whether blinded (i.e. the assessor is unaware of the study author's name, institution, sponsorship, journal, etc.) versus unblinded assessments of risk of bias yield systematically different assessments in a systematic review.

Objectives: To determine whether blinded versus unblinded assessments of risk of bias yield systematically different assessments in a systematic review.

Search strategy: We searched MEDLINE (1966 to September week 4 2009), CINAHL (1982 to May week 3 2008), All EBM Reviews (inception to 6 October 2009), EMBASE (1980 to 2009 week 40) and HealthStar (1966 to September week 4 2009) (all Ovid interface). We applied no restrictions regarding language of publication, publication status or study design. We examined reference lists of included studies and contacted experts for potentially relevant literature.

Selection criteria: We included any study that examined blinded versus unblinded assessments of risk of bias included within a systematic review.

Data collection and analysis: We extracted information from each of the included studies using a pre-specified 16-item form. We summarized the level of agreement between blinded and unblinded assessments of risk of bias descriptively. We calculated the standardized mean difference whenever possible.

Main results: We included six randomized controlled trials (RCTs). Four studies had unclear risk of bias and two had high risk of bias. The results of these RCTs were not consistent; two demonstrated no differences between blinded and unblinded assessments, two found that blinded assessments had significantly lower quality scores, and another observed significantly higher quality scores for blinded assessments. The remaining study did not report the level of significance. We pooled five studies reporting sufficient information in a meta-analysis. We observed no statistically significant difference in risk of bias assessments between blinded or unblinded assessments (standardized mean difference -0.13, 95% confidence interval -0.42 to 0.16). The mean difference might be slightly inaccurate, as we did not adjust for clustering in our meta-analysis. We observed inconsistency of results visually and noted statistical heterogeneity.

Authors' conclusions: Our review highlights that discordance exists between studies examining blinded versus unblinded risk of bias assessments at the systematic review level. The best approach to risk of bias assessment remains unclear, however, given the increased time and resources required to conceal reports effectively, it may not be necessary for risk of bias assessments to be conducted under blinded conditions in a systematic review.

PubMed Disclaimer

Conflict of interest statement

Kate Morissette: none declared.

Andrea C Tricco: none declared.

Tanya Horsley: none declared.

Maggie Hong Chen: none declared.

David Moher: author of an included study; one of the developers of the Cochrane 'Risk of bias' tool.

Figures

1
1
Study flow diagram.
2
2
Funnel plot of comparison: 1 Blinded versus unblinded, outcome: 1.1 Blinded versus unblinded.
1.1
1.1. Analysis
Comparison 1 Blinded versus unblinded, Outcome 1 Blinded versus unblinded.
See this image and copyright information in PMC

Update of

References

References to studies included in this review

Berard 2000 {published data only}
    1. Bèrard A, Andreu N, Tètrault J, Niyonsenga T, Myhal D. Reliability of Chalmers' scale to assess quality in meta‐analyses on pharmacological treatments for osteoporosis. Annals of Epidemiology 2000;10(8):498‐503. - PubMed
Berlin 1997 {published data only}
    1. Berlin JA for the University of Pennsylvania Meta‐analysis Blinding Study Group. Does blinding of readers affect the results of meta‐analyses?. Lancet 1997;350(9072):185‐6. - PubMed
Clark 1999 {published data only}
    1. Clark HD, Wells GA, Huït C, McAlister FA, Salmi LR, Fergusson D, et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Controlled Clinical Trials 1999;20(5):448‐52. - PubMed
Jadad 1996 {published data only}
    1. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Controlled Clinical Trials 1996;17(1):1‐12. - PubMed
Moher 1998 {published data only}
    1. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta‐analyses?. Lancet 1998;352(9128):609‐13. - PubMed
Verhagen 1998 {published data only}
    1. Verhagen AP, Vet HC, Bie RA, Kessels AG, Boers M, Knipschild PG. Balneotherapy and quality assessment: interobserver reliability of the Maastricht criteria list and the need for blinded quality assessment. Journal of Clinical Epidemiology 1998;51(4):335‐41. - PubMed

References to studies excluded from this review

van Tulder 1997 {unpublished data only}
    1. Tulder MW, Scholten RJPM, Koes BM, Bouter LM. Blinded assessment of the methodological quality. Cochrane Colloquium, Poster 266 1997.

Additional references

Assendelft 1999
    1. Assendelft WJ, Scholten RJ, Eijk JT, Bouter LM. The practice of systematic reviews. III. Evaluation of methodological quality of research studies. Nederlands Tijdschrift Geneeskunde 1999;143(14):714‐9. - PubMed
Chalmers 1981
    1. Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, et al. A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 1981;2(1):31‐49. - PubMed
Ellis 2010
    1. Ellis PD. The Essential Guide to Effect Sizes: An Introduction to Statistical Power, Meta‐Analysis and the Interpretation of Research Results. United Kingdom: Cambridge University Press, 2010.
Hartling 2009
    1. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009;339:b4012. - PMC - PubMed
Herbison 2006
    1. Herbison P, Hay‐Smith J, Gillespie WJ. Adjustment of meta‐analyses on the basis of quality scores should be abandoned. Journal of Clinical Epidemiology 2006;59(12):1249‐56. - PubMed
Higgins 2009
    1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009], The Cochrane Collaboration, 2009. Available from www.cochrane‐handbook.org.
Jüni 1999
    1. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta‐analysis. JAMA 1999;282(11):1054‐60. - PubMed
Landis 1977
    1. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159e74. - PubMed
Moher 1996
    1. Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. International Journal of Technology Assessment in Health Care 1996;12(2):195‐208. - PubMed
Moher 2007
    1. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4(3):e78. - PMC - PubMed
Plint 2006
    1. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Medical Journal of Australia 2006;185(5):263‐7. - PubMed
Sanderson 2007
    1. Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. International Journal of Epidemiology 2007;36:666‐76. - PubMed
Tricco 2008
    1. Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. Journal of Clinical Epidemiology 2008;61(5):422‐34. - PubMed
Wen 2008
    1. Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, et al. The reporting quality of meta‐analyses improves: a random sampling study. Journal of Clinical Epidemiology 2008;61(8):770‐5. - PubMed
West 2002
    1. West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute‐University of North Carolina Evidence‐based Practice Center under Contract No. 290‐97‐0011) 2002; Vol. AHRQ Publication No.02‐E016 2002. - PMC - PubMed

MeSH terms

LinkOut - more resources