Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Meta-Analysis
. 1998 Oct 31;317(7167):1185-90.
doi: 10.1136/bmj.317.7167.1185.

The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

R Kunz  1 , A D Oxman
Affiliations
Meta-Analysis

The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

R Kunz et al. BMJ. .

Abstract

Objective: To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres.

Design: Systematic review.

Selection criteria: Cohorts or meta-analyses of clinical trials that included an empirical assessment of the relation between randomisation and estimates of effect.

Data sources: Cochrane Review Methodology Database, Medline, SciSearch, bibliographies, hand searching of journals, personal communication with methodologists, and the reference lists of relevant articles.

Main outcome measures: Relation between randomisation and estimates of effect.

Results: Eleven studies that compared randomised controlled trials with non-randomised controlled trials (eight for evaluations of the same intervention and three across different interventions), two studies that compared trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relation between quality scores and estimates of treatment effects, were identified. Failure to use random allocation and concealment of allocation were associated with relative increases in estimates of effects of 150% or more, relative decreases of up to 90%, inversion of the estimated effect and, in some cases, no difference. On average, failure to use randomisation or adequate concealment of allocation resulted in larger estimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups.

Conclusions: Failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are. The size of these distortions can be as large as or larger than the size of the effects that are to be detected.

PubMed Disclaimer

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Cochrane AL. Effectiveness and efficiency: random reflections on health services. London: Nuffield Provincial Hospitals Trust; 1972. pp. 20–25.
    1. Committee for Evaluating Medical Technologies in Clinical Use. Assessing medical technologies. Washington DC: National Academy Press; 1985. pp. 76–78.
    1. US Congress; Office of Technology Assessment. Identifying health technologies that work: searching for evidence, OTA-H-608. Washington DC: US Government Printing Office; 1994. pp. 41–51.
    1. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312:1215–1218. - PMC - PubMed
    1. Weiss CH. Methods for studying programs and policies. 2nd ed. Upper Saddle River: Prentice Hall; 1998. Evaluation; pp. 229–233.

MeSH terms