Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Apr 13;2011(4):MR000012.
doi: 10.1002/14651858.MR000012.pub3.

Randomisation to protect against selection bias in healthcare trials

Jan Odgaard-Jensen  1 , Gunn E Vist, Antje Timmer, Regina Kunz, Elie A Akl, Holger Schünemann, Matthias Briel, Alain J Nordmann, Silvia Pregno, Andrew D Oxman
Affiliations

Randomisation to protect against selection bias in healthcare trials

Jan Odgaard-Jensen et al. Cochrane Database Syst Rev. .

Abstract

Background: Randomised trials use the play of chance to assign participants to comparison groups. The unpredictability of the process, if not subverted, should prevent systematic differences between comparison groups (selection bias). Differences due to chance will still occur and these are minimised by randomising a sufficiently large number of people.

Objectives: To assess the effects of randomisation and concealment of allocation on the results of healthcare studies.

Search strategy: We searched the Cochrane Methodology Register, MEDLINE, SciSearch and reference lists up to September 2009. In addition, we screened articles citing included studies (ISI Science Citation Index) and papers related to included studies (PubMed).

Selection criteria: Eligible study designs were cohorts of studies, systematic reviews or meta-analyses of healthcare interventions that compared random allocation versus non-random allocation or adequate versus inadequate/unclear concealment of allocation in randomised trials. Outcomes of interest were the magnitude and direction of estimates of effect and imbalances in prognostic factors.

Data collection and analysis: We retrieved and assessed studies that appeared to meet the inclusion criteria independently. At least two review authors independently appraised methodological quality and extracted information. We prepared tabular summaries of the results for each comparison and assessed the results across studies qualitatively to identify common trends or discrepancies.

Main results: A total of 18 studies (systematic reviews or meta-analyses) met our inclusion criteria. Ten compared random allocation versus non-random allocation and nine compared adequate versus inadequate or unclear concealment of allocation within controlled trials. All studies were at high risk of bias.For the comparison of randomised versus non-randomised studies, four comparisons yielded inconclusive results (differed between outcomes or different modes of analysis); three comparisons showed similar results for random and non-random allocation; two comparisons had larger estimates of effect in non-randomised studies than in randomised trials; and two comparisons had larger estimates of effect in randomised than in non-randomised studies.Five studies found larger estimates of effect in trials with inadequate concealment of allocation than in trials with adequate concealment. The four other studies did not find statistically significant differences.

Authors' conclusions: The results of randomised and non-randomised studies sometimes differed. In some instances non-randomised studies yielded larger estimates of effect and in other instances randomised trials yielded larger estimates of effect. The results of controlled trials with adequate and inadequate/unclear concealment of allocation sometimes differed. When differences occurred, most often trials with inadequate or unclear allocation concealment yielded larger estimates of effects relative to controlled trials with adequate allocation concealment. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non-random allocation or controlled trials with inadequate or unclear allocation concealment.

PubMed Disclaimer

Conflict of interest statement

JOJ is statistician with the Methodology Review Group.

Figures

1
1
Studies of randomised trials compared with non‐randomised trials across different interventions and conditions ‐ continuous estimates of effect
2
2
Studies of controlled trials with adequate concealment of allocation compared with inadequate/unclear concealment of allocation across different interventions and conditions ‐ ratio of odds ratios
See this image and copyright information in PMC

Update of

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

References to studies included in this review

Balk 2002 {published data only}
    1. Balk EM, Bonis PAL, Moskowitz H, Schmid CH, Ioannidis JPA, Wang C, et al. Correlation of quality measures with estimates of treatment effect in meta‐analysis of randomized controlled trials. JAMA 2002;287:2973‐82. - PubMed
Carroll 1996 {published data only}
    1. Carroll D, Tramer M, McQuay H, Nye B, Moore A. Randomization is important in studies with pain outcomes: systematic review of transcutaneous electrical nerve stimulation in acute postoperative pain.. British Journal of Anaesthesia 1996;77:798‐803. - PubMed
Chalmers 1983 {published data only}
    1. Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials.. New England Journal of Medicine 1983;309:1358‐61. - PubMed
Clifford 2002 {published data only}
    1. Clifford TJ, Barrowman NJ, Moher D. Funding source, trial outcome and reporting quality: are they related? Results of a pilot study. BMC Health Services Research 2002;2:18. - PMC - PubMed
Colditz 1989 {published data only}
    1. Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparisons of therapy. I: Medical. Statistics in Medicine 1989;8:441‐54. - PubMed
Egger 2003 {published data only}
    1. Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical Study. Health Technology Assessment 2003;7(1):1‐76. - PubMed
Hedrick 1989 {published data only}
    1. Hedrick SC, Koepsell TD, Inui T. Meta‐analysis on home‐care effects on mortality and nursing‐home placement. Medical Care 1989;27:1015‐26. - PubMed
Heinsman 1996 {published data only}
    1. Heinsman DT, Shadish WR. Assignment methods in experimentation: when do non‐randomised experiments approximate answers from randomised experiments?. Psychological Methods 1996;1:154‐69.
Kjaergard 2001 {published data only}
    1. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta‐analysis. Annals of Internal Medicine 2001;135:982‐9. - PubMed
Linde 1999 {published data only}
    1. Linde K, Clausius N, Ramirez G, Melchart D, Eitel F, Hedges LV, et al. Are the clinical effects of homoeopathy placebo effects? A meta‐analysis of placebo controlled trials. Lancet 1997;350:834‐43. - PubMed
    1. Linde K, Scholz M, Ramirez G, Clausius N, Melcart D, Jonas WB. Impact of study quality on outcome in placebo‐controlled trials of homeopathy. Journal of Clinical Epidemiology 1999;52:631‐6. - PubMed
Lipsey 1993 {published data only}
    1. Lipsey MW, Wilson DB. The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta‐analysis. American Psychologist 1993;48(12):1181‐209. - PubMed
Miller 1989 {published data only}
    1. Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II Surgical. Statistics in Medicine 1989;8:455‐66. - PubMed
Moher 1998 {published data only}
    1. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta‐analyses?. Lancet 1998;352(9128):609‐13. - PubMed
Ottenbacher 1991 {published data only}
    1. Ottenbacher KJ. Epistemology and experimentation: an examination of quality factors in research design. American Journal of Occupational Therapy 1991;45(10):917‐23. - PubMed
Ottenbacher 1992 {published data only}
    1. Ottenbacher K. Impact of random assignment on study outcome: an empirical examination. Controlled Clinical Trials 1992;13:50‐61. - PubMed
Schulz 1995 {published data only}
    1. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408‐12. - PubMed
Shadish 1996 {published data only}
    1. Shadish WR, Ragsdale K. Random versus nonrandom assignment in controlled experiments: do you get the same answer?. Journal of Consulting and Clinical Psychology 1996;64(6):1290‐305. - PubMed
Shang 2005 {published data only}
    1. Shang A, Huwiler‐Muntener K, Nartey L, Jüni P, Dorig S, Sterne JAC, et al. Are the clinical effects of homeopathy placebo effects? Comparative study of placebo‐controlled trials of homoepathy and allopathy. Lancet 2005;366:726‐32. - PubMed

References to studies excluded from this review

Aronson 1996 {published data only}
    1. Aronson R, Offman HJ, Joffe RT, Naylor CD. Triiodothyronine augmentation in the treatment of refractory depression. A meta‐analysis. Archives of General Psychiatry 1996;53(9):842‐8. - PubMed
Benson 2000 {published data only}
    1. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. New England Journal of Medicine 2000;342:1878‐86. - PubMed
Bhansali 1996 {published data only}
    1. Bhansali MS, Vaidya JS, Bhatt RG, Patil PK, Badwe RA, Desai PB. Chemotherapy for carcinoma of the esophagus: a comparison of evidence from meta‐analyses of randomized trials and of historical control studies. Annals of Oncology 1996;7(4):355‐9. - PubMed
Chalmers 1977 {published data only}
    1. Chalmers TC, Matta RJ, Smith H Jr, Kunzler AM. Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction. New England Journal of Medicine 1997;297:1091‐6. - PubMed
Diehl 1986 {published data only}
    1. Diehl LF, Perry DJ. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid?. Journal of Clinical Oncology 1986;4:1114‐20. - PubMed
Emerson 1990 {published data only}
    1. Emerson JD, Burdick E, Hoaglin DC, Mosteller F, Chalmers TC. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Controlled Clinical Trials 1990;11:339‐52. - PubMed
Forgie 1998 {published data only}
    1. Forgie MA, Wells PS, Laupacis A, Fergusson D. Preoperative autologous donation decreases allogeneic transfusion but increases exposure to all red blood cell transfusion: results of a meta‐analysis. International Study of Perioperative Transfusion (ISPOT) Investigators. Archives of Internal Medicine 1998;158(6):610‐6. - PubMed
Gilbert 1977 {published data only}
    1. Gilbert JP, McPeek B, Mosteller F. Statistics and ethics in surgery and anesthesia. Science 1977;198(4318):684‐9. - PubMed
Guyatt 2000 {published data only}
    1. Guyatt GH, DiCenso A, Farewell V, Willan A, Griffith L. Randomized trials versus observational studies in adolescent pregnancy prevention. Journal of Clinical Epidemiology 2000;53(2):167‐74. - PubMed
Hovell 1982 {published data only}
    1. Hovell MF. The experimental evidence for weight‐loss treatment of essential hypertension: a critical review. American Journal of Public Health 1982;72(4):359‐68. - PMC - PubMed
Hutchinson 1999 {published data only}
    1. Hutchinson BG, Oxman AD, Shannon HS, Lloyd S, Altmayer CA, Thomas K. Clinical effectiveness of pneumococcal vaccine. Meta‐analysis. Canadian Family Physician 1999;45:2381‐93. - PMC - PubMed
Imperiale 1990 {published data only}
    1. Imperiale TF, McCullough AJ. Do corticosteroids reduce mortality from alcoholic hepatitis? A meta analysis of the randomized trials. Annals of Internal Medicine 1990;113:299‐307. - PubMed
Ioannidis 2001 {published data only}
    1. Ioannidis JP, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA 2001;286:821‐30. - PubMed
Kasiske 1993 {published data only}
    1. Kasiske BL, Heim‐Duthoy K, Ma JZ. Elective cyclosporine withdrawal after renal transplantation. A meta‐analysis. JAMA 1993;269:395‐400. - PubMed
Kasiske 1998 {published data only}
    1. Kasiske BL, Lakatua JD, Ma JZ, Louis TA. A meta‐analysis of the effects of dietary protein restriction on the rate of decline in renal function. American Journal of Kidney Disease 1998;31:954‐61. - PubMed
Kerlikowske 1995 {published data only}
    1. Kerlikowske K, Grady D, Rubin SM, Sandrock C, Ernster VL. Efficacy of screening mammography. A meta‐analysis. JAMA 1995;273(2):149‐54. - PubMed
Khan 1996 {published data only}
    1. Khan KS, Daya S, Jadad A. The importance of quality of primary studies in producing unbiased systematic reviews. Archives of Internal Medicine 1996;156:661‐6. - PubMed
Koes 1994 {published data only}
    1. Koes BW, Tulder MW, Windt WM, Bouter LM. The efficacy of back schools: a review of randomized clinical trials. Journal of Clinical Epidemiology 1994;47(8):851‐62. - PubMed
Kownacki 1999 {published data only}
    1. Kownacki RJ, Shadish WR. Does Alcoholics Anonymous work? The results from a meta‐analysis of controlled experiments. Substance Use & Misuse 1999;34:1897‐916. - PubMed
MacArthur 1995 {published data only}
    1. Macarthur C, Foran PJ, Bailar JC 3rd. Qualitative assessment of studies included in a meta‐analysis: DES and the risk of pregnancy loss. Journal of Clinical Epidemiology 1995;48(6):739‐47. - PubMed
Mehta 1999 {published data only}
    1. Mehta RH, Bates ER. Coronary stent implantation in acute myocardial infarction. American Heart Journal 1999;137:603‐11. - PubMed
Moher 1999 {published data only}
    1. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta‐analysis. Health Technology Assessment 1999;3(12):100p. - PubMed
Morrison 1997 {published data only}
    1. Morrison B, Lilford RJ, Earnst E. Methodological rigor and results of clinical trials of homeopathic remedies. Perfusion 2000;13:132‐8.
Mullen 1997 {published data only}
    1. Mullen PD, Simons‐Morton DG, Ramirez G, Frankowski RF, Green LW, Mains DA. A meta‐analysis of trials evaluating patient education and counseling for three groups of preventive health behaviors. Patient Education and Counseling 1997;32(3):157‐73. - PubMed
Nurmohamed 1992 {published data only}
    1. Nurmohamed MT, Rosendaal FR, Buller HR, Dekker E, Hommes DW, Vandenbroucke JP, et al. Low molecular weight heparin versus standard heparin in general and orthopaedic surgery: a meta‐analysis. Lancet 1992;340:152‐6. - PubMed
Ortiz 1998 {published data only}
    1. Ortiz Z, Shea B, Suarez Almazor ME, Moher D, Wells GA, Tugwell P. The efficacy of folic acid and folinic acid in reducing methotrexate gastrointestinal toxicity in rheumatoid arthritis. A meta‐analysis of randomized controlled trials. Journal of Rheumatology 1998;25:36‐43. - PubMed
Ottenbacher 1993 {published data only}
    1. Ottenbacher KJ, Jannell S. The results of clinical trials in stroke rehabilitation research. Archives of Neurology 1993;50(1):37‐44. - PubMed
Pagnin 2004 {published data only}
    1. Pagnin D, Queiroz V, Pini S, Cassano GB. Efficacy of ECT in depression: a meta‐analytic review. Journal of ECT 2004;20:13‐20. - PubMed
Potter 1998 {published data only}
    1. Potter J, Langhorne P, Roberts M. Routine protein energy supplementation in adults: systematic review. BMJ 1998;317:495‐501. - PMC - PubMed
Pyorala 1995 {published data only}
    1. Pyorala S, Huttunen NP, Uhari M. A review and meta‐analysis of hormonal treatment of cryptorchidism. Journal of Clinical Endocrinology & Metabolism 1995;80:2795‐9. - PubMed
RMIT Group 1994 {published data only}
    1. Recurrent Miscarriage Immunotherapy Trialists Group. Worldwide collaborative observational study and meta analysis on allogenic leukocyte immunotherapy for recurrent spontaneous abortion. American Journal of Reproductive Immunology 1994;32:55‐72. - PubMed
Rozenberg 1999 {published data only}
    1. Rozenberg S, Dubourg G, Khalifa P, Paolozzi L, Maheu E, Ravaud P. Efficacy of epidural steroids in low back pain and sciatica. A critical appraisal by a French Task Force of randomized trials. Critical Analysis Group of the French Society for Rheumatology. Revue de Rhumatisme (English edition) 1999;66(2):79‐85. - PubMed
Sacks 1982 {published data only}
    1. Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. American Journal of Medicine 1982;72:233‐40. - PubMed
Shadish 1997 {published data only}
    1. Shadish WR, Heinsman DT. Experiments versus quasi‐experiments: do they yield the same answer?. In: Bukoski WJ editor(s). Meta‐analysis of Drug Abuse Prevention Programs. Washington DC: US Department of Health and Human Services, National Institutes of Health, 1997:147‐64. - PubMed
Shadish 2001 {published data only}
    1. Shadish WR, Glaser RR. Differences between randomised and nonrandomised experiments using meta‐analysis: a methodological note. 2001.
Stanton 1997 {published data only}
    1. Stanton MD, Shadish WR. Outcome, attrition and family‐couples treatment for drug abuse. Psychological Bulletin 1997;122:170‐91. - PubMed
Stieb 1990 {published data only}
    1. Stieb D, Frayha HH, Oxman AD, Shannon HS, Hutchison BG, Crombie F. The effectiveness and usefulness of Haemophilus influenzae type B vaccines: a systematic overview (meta‐analysis). Canadian Medical Association Journal 1990;142:719‐32. - PMC - PubMed
Watson 1994 {published data only}
    1. Watson A, Vandekerckhove P, Lilford R, Vail A, Brosens I, Hughes E. A meta‐analysis of the therapeutic role of oil soluble contrast media at hysterosalpingography: a surprising result?. Fertility and Sterility 1994;61:470‐7. - PubMed
Weisburd 2001 {published data only}
    1. Weisburd D, Lum CM, Petrosino A. Does research design affect study outcomes in criminal justice. Annals of the American Academy of Political and Social Science 2001;578:50‐70.
Wortman 1983 {published data only}
    1. Wortman P, Yeaton WH. Synthesis of results in trials of coronary artery bypass graft surgery. In: Light R editor(s). Evaluation Studies Review Annual. Vol. 8, Beverley Hills, CA: Sage Publications, 1983:536‐57.

References to studies awaiting assessment

Abraham 2010 {published data only}
    1. Abraham NS, Byrne CJ, Young JM, Solomon MJ. Meta‐analysis of well‐designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials. Journal of Clinical Epidemiology 2010;63(3):238‐45. - PubMed
Bellavite 2006 {published data only}
    1. Bellavite P, Ortolani R, Pontarollo F, Piasere V, Benato G, Conforti A. Immunology and homeopathy. 4. Clinical studies ‐ Part 1. Evidence‐Based Complementary and Alternative Medicine 2006;3:293‐301. - PMC - PubMed
Bhandari 2004 {published data only}
    1. Bhandari M, Tornetta P, Ellis T, Audige L, Sprague S, Kuo JC, et al. Hierarchy of evidence: differences in results between non‐randomized studies and randomized trials in patients with femoral neck fractures. Archives of Orthopaedic and Trauma Surgery 2004;124:10‐6. - PubMed
Bhogal 2005 {published data only}
    1. Bhogal SK, Foley N, Teasell R, Speechley M. Methodological differences influencing treatment effect [abstract]. American Journal of Epidemiology 2005;161:S67.
Britton 1998 {published data only}
    1. Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Choosing between randomised and non‐randomised studies: a systematic review. Health Technology Assessments 1998;2:No.13. - PubMed
Cipriani 2006 {published data only}
    1. Cipriani A, Malvini L, Furukawa T, Barbui C. Does the quality of antidepressant trials affect outcome estimates? [abstract]. XIV Cochrane Colloquium; 2006 October 23‐26; Dublin, Ireland 2006; Vol. 169.
Cipriani 2007 {published data only}
    1. Cipriani A, Malvini L, Furukawa TA, Barbui C. Relationship between quality of reports of antidepressant randomized controlled trials and treatment estimates: systematic review, meta‐analysis, and meta‐regression analysis. Journal of Clinical Psychopharmacology 2007;27:352‐6. - PubMed
Concato 2000 {published data only}
    1. Concato J, Shah N, Horwitz RI. Randomized controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine 2000;342:1887‐92. - PMC - PubMed
Fenwick 2008 {published data only}
    1. Fenwick J, Needleman I, Moles D. Effect of allocation concealment and examiner masking on magnitude of clinical outcomes. Poster presentation at the 16th Cochrane Colloquium: Evidence in the era of globalisation; 2008 Oct 3‐7; Freiburg, Germany [abstract]. Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen 2008;102:59. - PubMed
Fenwick 2008a {published data only}
    1. Fenwick J, Needleman IG, Moles DR. The effect of bias on the magnitude of clinical outcomes in periodontology: a pilot study. Journal of Clinical Periodontology 2008;35:775‐82. - PubMed
Fernandez‐de‐las‐Penas 06 {published data only}
    1. Fernandez‐de‐las‐Penas C, Alonso‐Blanco C, San Roman J, Miangolarra‐Page JC. Methodological quality of randomized controlled trials of spinal manipulation and mobilization in tension‐type headache, migraine, and cervicogenic headache. Journal of Orthopaedic & Sports Physical Therapy 2006;36:160‐9. - PubMed
Ferriter 2005 {published data only}
    1. Ferriter M, Huband N. Does the non‐randomized controlled study have a place in the systematic review? A pilot study. Criminal Behaviour and Mental Health 2005;15:111‐20. - PubMed
Foley 2005 {published data only}
    1. Foley N, Speechley M, Salter K, Bhogal S, Jutai J, Teasell R. Concealed allocation: an under‐reported and misunderstood component of trial methodology in stroke rehabilitation [abstract]. American Journal of Epidemiology 2005;161:S67.
Furlan 2008a {published data only}
    1. Furlan AD, Tomlinson G, Jadad AA, Bombardier C. Examining heterogeneity in meta‐analysis: comparing results of randomized trials and nonrandomized studies of interventions for low back pain. Spine 2008;33:339‐48. - PubMed
Furlan 2008b {published data only}
    1. Furlan AD, Tomlinson G, Jadad AA, Bombardier C. Methodological quality and homogeneity influenced agreement between randomized trials and nonrandomized studies of the same intervention for back pain. Journal of Clinical Epidemiology 2008;61:209‐31. - PubMed
Gluud 2008 {published data only}
    1. Gluud LL, Thorlund K, Gluud C, Woods L, Harris R, Sterne JA. Correction: reported methodologic quality and discrepancies between large and small randomized trials in meta‐analyses. Annals of Internal Medicine 2008;149:219. - PubMed
Grossarth‐Maticek 2006 {published data only}
    1. Grossarth‐Maticek R, Ziegler R. Randomised and non‐randomised prospective controlled cohort studies in matched‐pair design for the long‐term therapy of breast cancer patients with a mistletoe preparation (Iscador): a re‐analysis. European Journal of Medical Research 2006;11:485‐95. - PubMed
Hartz 2005 {published data only}
    1. Hartz A, Bentler S, Charlton M, Lanska D, Butani Y, Soomro GM, et al. Assessing observational studies of medical treatments. Emerging Themes in Epidemiology 2005;2:8. - PMC - PubMed
Henry 2001 {published data only}
    1. Henry DM. Agreement between randomized and non‐randomized studies ‐ the effects of bias and confounding. 9th Annual Cochrane Colloquium Abstracts, October 2001 in Lyon 2001.
Jin 2003 {published data only}
    1. Jin SL, Wang YP, Zhao PC, Guo Z, Yang JL. Analysis of randomized controlled trials/clinical controlled trials on chronic gastritis in China. Chinese Journal of Evidence Based Medicine 2003;3:22‐5.
Kelly 2001 {published data only}
    1. Kelly AK. The last word in trial quality? The impact of selection and performance bias within a series of integrated systematic reviews. 9th Annual Cochrane Colloquium Abstracts, October 2001 in Lyon 2001.
Khan 1996b {published data only}
    1. Khan KS, Daya SD, Collins JA, Walter SD. Empirical evidence of bias in infertility research: overestimation of treatment effect in crossover trials using pregnancy as the outcome measure. Fertility and Sterility 1996;65:939‐45. - PubMed
Kitsios 2009 {published data only}
    1. Kitsios G, Zintzaras E. ACE (I/D) polymorphism and response to treatment in coronary artery disease: a comprehensive database and meta‐analysis involving study quality evaluation. BMC Medical Genetics 2009;10:50. - PMC - PubMed
Kjaergaard 2008 {published data only}
    1. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta‐analyses (vol 135, pg982, 2001). Annals of Internal Medicine 2008;149:219. - PubMed
Kyriakidi 2002 {published data only}
    1. Kyriakidi M, Ioannidis JPA. Design and quality considerations for randomized controlled trials in systemic sclerosis. Arthritis Care & Research 2002;47:73‐81. - PubMed
Linde 1999b {published data only}
    1. Linde K, Melchart D, Scholz M. Should systematic reviews include non‐randomized trials? Quality and results of various study types investigating the effect or acupuncture on idiopathic headaches. 7th Annual Cochrane Colloquium Abstracts, October 1999 in Rome 1999.
Manzoli 2007 {published data only}
    1. Manzoli L, Schioppa F, Boccia A, Villari P. The efficacy of influenza vaccine for healthy children: a meta‐analysis evaluating potential sources of variation in efficacy estimates including study quality. Pediatric Infectious Disease Journal 2007;26:97‐106. - PubMed
Papanikolaou 2006 {published data only}
    1. Papanikolaou PN, Christidi GD, Ioannidis JPA. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. Canadian Medical Association Journal 2006;174(5):635‐41. - PMC - PubMed
Pildal 2007 {published data only}
    1. Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta‐analyses of randomized trials. International Journal of Epidemiology 2007;36:847‐57. - PubMed
Reimold 1992 {published data only}
    1. Reimold SC, Chalmers TC, Berlin JA, Antman EM. Assessment of the efficacy and safety of antiarrhythmic therapy for chronic atrial fibrillation: observations on the role of trial design and implications of drug related mortality. American Heart Journal 1992;124:924‐32. - PubMed
Rodgers 2008 {published data only}
    1. Rodgers M, Chambers D, Woolacott N. To randomise or not to randomise: a matter of perspective? [abstract]. Fifth Annual Meeting HTAi; 2008 July 6‐9; Montreal, Canada. 2008; Vol. 41.
Sacks 1983 {published data only}
    1. Sacks HS, Chalmers TC, Smith H. Sensitivity and specificity of clinical trials. Randomized v historical controls. Archives of Internal Medicine 1983;143:753‐5. - PubMed
Savovic 2008 {published data only}
    1. Savovic J, Harris RJ, The BC. The association of three bias domains with the treatment effect estimates in randomised control trials: combined analysis of meta‐epidemiological studies. Oral presentation at the 16th Cochrane Colloquium: Evidence in the era of globalisation; 2008 Oct 3‐7; Freiburg, Germany [abstract]. Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen 2008;102:29‐30.
Shadish 2008 {published data only}
    1. Shadish WR, Clark MH, Steiner PM. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments.. Journal of the American Statistical Association 2008;103:1334‐43.
Shikata 2008 {published data only}
    1. Shikata S, Nakayama T, Yamagishi H. Quality of surgical randomized controlled trials for acute cholecystitis: assessment based on CONSORT and additional check items. Journal of Hepato‐Biliary‐Pancreatic Surgery 2008;15:297‐303. - PubMed
Siersma 2007 {published data only}
    1. Siersma V, Als‐Nielsen B, Chen W, Hilden J, Gluud LL, Gluud C. Multivariable modelling for meta‐epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Statistics in Medicine 2007;26:2745‐58. - PubMed
Singh 2009 {published data only}
    1. Singh JA, Murphy S, Bhandari M. Trial sample size, but not trial quality, is associated with positive study outcome. Journal of Clinical Epidemiology 2010;63(2):154‐62. - PMC - PubMed
Verhagen 2002 {published data only}
    1. Verhagen AP, Vet HCW, Vermer F, Widdershoven JWMG, Bie RA, Kessels AG, et al. The influence of methodologic quality on the conclusion of a landmark meta‐analysis on thrombolytic therapy. International Journal of Technology Assessment in Health Care 2002;18:11‐23. - PubMed
Verhagen 2008 {published data only}
    1. Verhagen AP, Vet HCW, Willemsen S, Stijnen T. A meta‐regression analysis shows no impact of design characteristics on outcome in trials on tension‐type headaches. Journal of Clinical Epidemiology 2008;61:813‐8. - PubMed
Villari 2004 {published data only}
    1. Villari P, Manzoli L, Boccia A. Methodological quality of studies and patient age as major sources of variation in efficacy estimates of influenza vaccination in healthy adults: a meta‐analysis. Vaccine 2004;22:3475‐86. - PubMed
Wahlbeck 2000 {published data only}
    1. Wahlbeck K, Tuunainen A, Gilbody S, Adams CE. Influence of methodology on outcomes of randomised Clozapine trials. Pharmacopsychiatry 2000;33:54‐9. - PubMed
Wang 2009 {published data only}
    1. Wang L, Li YM, Li L. Meta‐analysis of randomized and controlled treatment trials for achalasia. Digestive Diseases and Sciences 2009;54(11):2303‐11. - PubMed
Wood 2006 {published data only}
    1. Wood L, Egger M, Gluud LL, Schulz K, Altman D, Jüni P, et al. The association of allocation concealment and blinding with estimated treatment effect varies according to type of outcome: a combined analysis of meta‐epidemiological studies [abstract]. XIV Cochrane Colloquium; 2006 October 23‐26; Dublin, Ireland 2006; Vol. 52.
Wood 2008 {published data only}
    1. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta‐epidemiological study. BMJ 2008;336:601‐5. - PMC - PubMed

Additional references

Austin 1992
    1. Austin SC, Stolley PD, Lasky T. The history of malariotherapy for neurosyphilis. JAMA 1992;268:516‐9. - PubMed
Black 1996
    1. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312:1215‐8. - PMC - PubMed
Britton 1999
    1. Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Three systematic reviews ‐ not so different answers. BMJ Electronic letters 29 September 1999.
Chalmers 1997
    1. Chalmers I. Assembling comparison groups to assess the effects of health care. Journal of the Royal Society of Medicine 1997;90:379‐86. - PMC - PubMed
CMRG Module
    1. The Editorial Team. Cochrane Methodology Review Group. About The Cochrane Collaboration (Cochrane Review Groups (CRGs)) 2005, Issue 2:Art. No.: METHOD.
Cochrane 1972
    1. Cochrane AL. Effectiveness and Efficiency: Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust, 1972:20‐5.
Counsell 1994
    1. Counsell CE, Clarke MJ, Slattery J, Sandercock PAG. The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis?. BMJ 1994;309:1677‐81. - PMC - PubMed
Dans 2002
    1. Dans A, McAlister F, Dans L, Richardson WS, Straus S, Guyatt G. Applying the results to individual patients. In: Guyatt G, Rennie D editor(s). Users' Guides to the Medical Literature. Chicago: AMA Press, 2002:369‐84.
Dickersin 1993
    1. Dickersin K, Min YI. NIH clinical trials and publication bias. Online Journal of Current Clinical Trials 1993;Doc No 50:4967. - PubMed
Dickersin 1997
    1. Dickersin K. How important is publication bias? A synthesis of available data. AIDS Education and Prevention 1997;9(1 Suppl):15‐21. - PubMed
EC/IC Bypass 1985
    1. The EC/IC Bypass Study Group. Failure of extracranial‐intracranial arterial bypass to reduce the risk of ischemic stroke. Results of an international randomized trial. New England Journal of Medicine 1985;313:1191‐200. - PubMed
Echt 1991
    1. Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. New England Journal of Medicine 1991;324:781‐8. - PubMed
Eclampsia 1995
    1. Eclampsia Trial Collaborative Group. Which anticonvulsant for women with eclampsia?. Lancet 1995;345:1455‐63. - PubMed
Egger 1997
    1. Egger M, Davey SG, Schneider M, Minder C. Bias in meta‐analysis detected by a simple, graphical test. BMJ 1997;315:629‐34. - PMC - PubMed
Gubler 1971
    1. Gubler A. Alternating hemiplegia, a sign of pontine lesion, and documentation of the proof of the facial decussation. In: Wolf JK editor(s). The Classical Brainstem Syndromes. Springfield: Charles C Thomas, 1971:9‐13.
Guyatt 2002
    1. Guyatt GH, Cook D, Devereaux PJ, Meade M, Straus S. Therapy. Users' guides to the medical literature. Chicago: AMA Press, 2002:55‐79.
Hopewell 2001
    1. Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database of Methodology Reviews 2001, Issue 3. [DOI: 10.1002/14651858.MR000011] - DOI - PMC - PubMed
Hopewell 2006
    1. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta‐analyses of randomized trials of health care interventions. The Cochrane Database of Methodology Reviews 2006, Issue 2. [DOI: 10.1002/14651858.MR000010.pub2] - DOI - PMC - PubMed
Jüni 1999
    1. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta‐analysis. JAMA 1999;282(11):1054‐60. - PubMed
Jüni 2001
    1. Jüni P, Altman DG, Egger M. Assessing the quality of randomised controlled trials. In: Egger M, Smith GD, Altman D editor(s). Systematic Reviews in Health Care: Meta‐Analysis in Context. London: BMJ Books, 2001.
Kleijnen 1997
    1. Kleijnen J, Gøtzsche P, Kunz RH, Oxman AD, Chalmers I. So what's so special about randomisation?. In: Maynard A, Chalmers I editor(s). Non‐random Reflections on Health Services Research: on the 25th Anniversary of Archie Cochrane's Effectiveness and Efficiency. London: BMJ Books, 1997:93‐106.
Kunz 1999
    1. Kunz R, Oxman AD. Two systematic reviews ‐ two different answers?. BMJ Electronic Letters 28 August 1999.
McKee 1999
    1. McKee M, Britton A, Black N, Mcpherson K, Sanderson C, Bain C. Methods in health services research. Interpreting the evidence: choosing between randomised and non‐randomised studies. BMJ 1999;319:312‐5. - PMC - PubMed
Moore 1995
    1. Moore TJ. Deadly Medicine. New York: Simon and Schuster, 1995.
MRC 1948
    1. Medical Research Council. Streptomycin treatment of pulmonary tuberculosis. BMJ 1948;ii:769‐82. - PMC - PubMed
Pockock 2000
    1. Pockock SJ, Elbourne DR. Randomised trials or observational tribulations?. New England Journal of Medicine 2000;324:1907‐9. - PubMed
Reeves 1998
    1. Reeves BC, MacLehose RR, Harvey IM, Sheldon TA, Russell IT, Black AMS. Comparison of effect size estimates derived from randomised and non‐randomised studies. In: Black N, Brazier J, Fitzpatrick R, Reeves B editor(s). Health Services Research Methods: A Guide to Best Practice. London: BMJ Publishing Group, 1998:73‐85.
Rothwell 2005
    1. Rothwell PM. External validity of randomised controlled trials: “To whom do the results of this trial apply?”. Lancet 2005;365:82‐93. - PubMed
Song 2000
    1. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technology Assessment 2000;4(10):1‐115. - PubMed
Teo 1993
    1. Teo KK, Yusuf S, Furberg CD. Effects of prophylactic antiarrhythmic drug therapy in acute myocardial infarction. An overview of results from randomized trials. JAMA 1993;270:1589‐95. - PubMed
US Office HTA 1994
    1. US Congress, Office of Technology Assessment. Identifying Health Technologies that Work: Searching for Evidence. OTA‐H‐608. Washington DC: US Government Printing Office, 1994:41‐51.
Valenstein 1986
    1. Valenstein ES. Great and Desperate Cures: The Rise and Decline of Psychosurgery and Other Radical Treatments for Mental Illness. New York: Basic Books, 1986.
Weiss 1998
    1. Weiss CH. Evaluation. Methods for Studying Programs and Policies. 2nd Edition. Upper Saddle River: Prentice Hall, 1998:229‐33.

References to other published versions of this review

Kunz 1998
    1. Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non‐randomised clinical trials. BMJ 1998;317(7167):1185‐90. - PMC - PubMed
Kunz 2002
    1. Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Methodology Reviews 2002, Issue 4. [DOI: 10.1002/14651858.MR000012] - DOI - PMC - PubMed

Publication types

MeSH terms