Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2008 Mar 15;336(7644):601-5.
doi: 10.1136/bmj.39465.451748.AD. Epub 2008 Mar 3.

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Lesley Wood  1 , Matthias Egger, Lise Lotte Gluud, Kenneth F Schulz, Peter Jüni, Douglas G Altman, Christian Gluud, Richard M Martin, Anthony J G Wood, Jonathan A C Sterne
Affiliations

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Lesley Wood et al. BMJ. .

Abstract

Objective: To examine whether the association of inadequate or unclear allocation concealment and lack of blinding with biased estimates of intervention effects varies with the nature of the intervention or outcome.

Design: Combined analysis of data from three meta-epidemiological studies based on collections of meta-analyses.

Data sources: 146 meta-analyses including 1346 trials examining a wide range of interventions and outcomes.

Main outcome measures: Ratios of odds ratios quantifying the degree of bias associated with inadequate or unclear allocation concealment, and lack of blinding, for trials with different types of intervention and outcome. A ratio of odds ratios <1 implies that inadequately concealed or non-blinded trials exaggerate intervention effect estimates.

Results: In trials with subjective outcomes effect estimates were exaggerated when there was inadequate or unclear allocation concealment (ratio of odds ratios 0.69 (95% CI 0.59 to 0.82)) or lack of blinding (0.75 (0.61 to 0.93)). In contrast, there was little evidence of bias in trials with objective outcomes: ratios of odds ratios 0.91 (0.80 to 1.03) for inadequate or unclear allocation concealment and 1.01 (0.92 to 1.10) for lack of blinding. There was little evidence for a difference between trials of drug and non-drug interventions. Except for trials with all cause mortality as the outcome, the magnitude of bias varied between meta-analyses.

Conclusions: The average bias associated with defects in the conduct of randomised trials varies with the type of outcome. Systematic reviewers should routinely assess the risk of bias in the results of trials, and should report meta-analyses restricted to trials at low risk of bias either as the primary analysis or in conjunction with less restrictive analyses.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Figures

None
Fig 1 Ratios of odds ratios comparing estimates of intervention effects in 532 trials with inadequate or unclear allocation concealment versus 272 trials with adequate concealment. Columns on the right show P values from tests of interaction, and estimates of the variability between meta-analyses in the size of bias
None
Fig 2 Ratios of odds ratios comparing intervention effect estimates in 314 non-blinded trials versus 432 blinded trials. Columns on the right show P values from tests of interaction, and estimates of the variability between meta-analyses in the size of bias
See this image and copyright information in PMC

References

    1. Chalmers I, Altman DG. Systematic reviews London: BMJ Publishing Group, 1995
    1. Gluud LL. Bias in clinical intervention research. Am J Epidemiol 2006;163:493-501. - PubMed
    1. Naylor CD. Meta-analysis and the meta-epidemiology of clinical research. BMJ 1997;315:617-9. - PMC - PubMed
    1. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12. - PubMed
    1. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609-13. - PubMed

Publication types