Empirical evidence of observer bias in randomized clinical trials: updated and expanded analysis of trials with both blinded and non-blinded outcome assessors

Abstract

Objectives: To study the impact of lack of blinding of outcome assessors on estimated treatment effects of randomized clinical trials.

Study design and setting: Meta-epidemiological study. We included randomized trials with binary or measurement scale outcomes that (1) allocated patients to subtrials with and without blinded outcome assessment, or (2) had both blinded and non-blinded assessments of the same outcome. We identified trials from previous meta-epidemiological studies and searched six databases from 2013 to 2024. We calculated a ratio of odds ratios (ROR) for each trial. A ROR < 1 indicated a more favorable effect estimate by the non-blinded assessor. We pooled RORs using random effects meta-analysis and conducted meta-regression, subgroup, and sensitivity analyses.

Results: We included 66 trials (9451 patients) across 18 clinical specialties. The pooled RORs in 43 trials (7055 patients) was 0.71 (0.55-0.92). Thirty of the 43 trials assessed highly subjective outcomes. Meta-regression showed no statistically significant association between ROR and scores for outcome subjectivity (P = .53), vulnerability (P = .91), and involvement (P = .99). Heterogeneity was partly explained by a larger impact in non-drug trials, ROR 0.62 (0.46-0.84), and industry-funded trials, ROR 0.57 (0.37-0.88). Sensitivity analyses, including imputed data for 23 trials (2396 patients randomized), did not modify the observed impact importantly.

Conclusion: We provide empirical evidence of considerable bias in effect estimates of randomized trials with non-blinded assessors of subjective binary and measurement scale outcomes. Non-blinded assessors exaggerated effect estimates, expressed as odds ratios, by 29% (8%-45%) on average. This strongly supports blinding outcome assessors of subjective outcomes.

Plain language summary: In a randomized clinical trial, the person evaluating the results (assessor) may be either unaware of the intervention received by participants (blinded assessor) or aware of it (non-blinded assessor). Knowing which treatment a patient received can influence the assessor's evaluation of the effect; for example, if an assessor has high expectations for a new experimental intervention, they may rate a patient's improvement more favorably in the group that received the intervention compared to the group that did not. We call this observer bias. In this study, we compared the results obtained from blinded assessors to those from non-blinded assessors within the same trials to estimate the impact of observer bias in randomized trials. We found that non-blinded assessors exaggerated the experimental intervention effect by approximately 29%, on average, compared to blinded assessors. Our results indicate that when an evaluation of a patient in a randomized trial requires judgment, there is potential for substantial bias if assessors are not blinded. To ensure more reliable results, randomized clinical trials should blind assessors whenever possible.

Keywords: Bias; Blinding; Empirical evidence; Meta-epidemiological studies; Observer bias; Randomized controlled trials as topic.