Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2014 Apr 29;2014(4):MR000034.
doi: 10.1002/14651858.MR000034.pub2.

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials

Andrew Anglemyer  1 , Hacsi T Horvath, Lisa Bero
Affiliations
Comparative Study

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials

Andrew Anglemyer et al. Cochrane Database Syst Rev. .

Update in

Abstract

Background: Researchers and organizations often use evidence from randomized controlled trials (RCTs) to determine the efficacy of a treatment or intervention under ideal conditions. Studies of observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and modifications of existing designs, including both randomized and observational, are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population.A systematic analysis of study design features, risk of bias, parameter interpretation, and effect size for all types of randomized and non-experimental observational studies is needed to identify specific differences in design types and potential biases. This review summarizes the results of methodological reviews that compare the outcomes of observational studies with randomized trials addressing the same question, as well as methodological reviews that compare the outcomes of different types of observational studies.

Objectives: To assess the impact of study design (including RCTs versus observational study designs) on the effect measures estimated.To explore methodological variables that might explain any differences identified.To identify gaps in the existing research comparing study designs.

Search methods: We searched seven electronic databases, from January 1990 to December 2013.Along with MeSH terms and relevant keywords, we used the sensitivity-specificity balanced version of a validated strategy to identify reviews in PubMed, augmented with one term ("review" in article titles) so that it better targeted narrative reviews. No language restrictions were applied.

Selection criteria: We examined systematic reviews that were designed as methodological reviews to compare quantitative effect size estimates measuring efficacy or effectiveness of interventions tested in trials with those tested in observational studies. Comparisons included RCTs versus observational studies (including retrospective cohorts, prospective cohorts, case-control designs, and cross-sectional designs). Reviews were not eligible if they compared randomized trials with other studies that had used some form of concurrent allocation.

Data collection and analysis: In general, outcome measures included relative risks or rate ratios (RR), odds ratios (OR), hazard ratios (HR). Using results from observational studies as the reference group, we examined the published estimates to see whether there was a relative larger or smaller effect in the ratio of odds ratios (ROR).Within each identified review, if an estimate comparing results from observational studies with RCTs was not provided, we pooled the estimates for observational studies and RCTs. Then, we estimated the ratio of ratios (risk ratio or odds ratio) for each identified review using observational studies as the reference category. Across all reviews, we synthesized these ratios to get a pooled ROR comparing results from RCTs with results from observational studies.

Main results: Our initial search yielded 4406 unique references. Fifteen reviews met our inclusion criteria; 14 of which were included in the quantitative analysis.The included reviews analyzed data from 1583 meta-analyses that covered 228 different medical conditions. The mean number of included studies per paper was 178 (range 19 to 530).Eleven (73%) reviews had low risk of bias for explicit criteria for study selection, nine (60%) were low risk of bias for investigators' agreement for study selection, five (33%) included a complete sample of studies, seven (47%) assessed the risk of bias of their included studies,Seven (47%) reviews controlled for methodological differences between studies,Eight (53%) reviews controlled for heterogeneity among studies, nine (60%) analyzed similar outcome measures, and four (27%) were judged to be at low risk of reporting bias.Our primary quantitative analysis, including 14 reviews, showed that the pooled ROR comparing effects from RCTs with effects from observational studies was 1.08 (95% confidence interval (CI) 0.96 to 1.22). Of 14 reviews included in this analysis, 11 (79%) found no significant difference between observational studies and RCTs. One review suggested observational studies had larger effects of interest, and two reviews suggested observational studies had smaller effects of interest.Similar to the effect across all included reviews, effects from reviews comparing RCTs with cohort studies had a pooled ROR of 1.04 (95% CI 0.89 to 1.21), with substantial heterogeneity (I(2) = 68%). Three reviews compared effects of RCTs and case-control designs (pooled ROR: 1.11 (95% CI 0.91 to 1.35)).No significant difference in point estimates across heterogeneity, pharmacological intervention, or propensity score adjustment subgroups were noted. No reviews had compared RCTs with observational studies that used two of the most common causal inference methods, instrumental variables and marginal structural models.

Authors' conclusions: Our results across all reviews (pooled ROR 1.08) are very similar to results reported by similarly conducted reviews. As such, we have reached similar conclusions; on average, there is little evidence for significant effect estimate differences between observational studies and RCTs, regardless of specific observational study design, heterogeneity, or inclusion of studies of pharmacological interventions. Factors other than study design per se need to be considered when exploring reasons for a lack of agreement between results of RCTs and observational studies. Our results underscore that it is important for review authors to consider not only study design, but the level of heterogeneity in meta-analyses of RCTs or observational studies. A better understanding of how these factors influence study effects might yield estimates reflective of true effectiveness.

PubMed Disclaimer

Conflict of interest statement

None to declare.

Figures

1
1
Flow chart depicting screening process
2
2
'Risk of bias' graph: review authors' judgements about each risk of bias item presented as percentages across all included studies.
3
3
'Risk of bias' summary: review authors' judgements about each risk of bias item for each included study.
4
4
Forest plot of comparison: 1 RCT vs Observational, outcome: 1.2 Pooled Ratio of Odds Ratios‐‐Study Design.
5
5
Forest plot of comparison: 1 RCT vs Observational, outcome: 1.3 Pooled Ratio of Odds Ratios‐‐Heterogeneity Subgroups.
6
6
Forest plot of comparison: 1 RCT vs Observational, outcome: 1.4 Pooled Ratio of Odds Ratios‐‐Pharmacological Studies Subgroups.
1.1
1.1. Analysis
Comparison 1 RCT vs Observational, Outcome 1 Summary Ratios of Ratios: RCTs vs Observational Studies.
1.2
1.2. Analysis
Comparison 1 RCT vs Observational, Outcome 2 Summary Ratios of Ratios: RCTs vs Observational Studies (Heterogeneity Subgroups).
1.3
1.3. Analysis
Comparison 1 RCT vs Observational, Outcome 3 Summary Ratios of Ratios: RCTs vs Observational Studies (Pharmacological Studies vs non‐Pharmacological Studies).
1.4
1.4. Analysis
Comparison 1 RCT vs Observational, Outcome 4 Summary Ratios of Ratios: RCTs vs Observational Studies (Propensity Scores).
See this image and copyright information in PMC

References

References to studies included in this review

Benson 2000 {published data only}
    1. Benson K, Hartz A. A comparison of observational studies and randomized, controlled trials. New England Journal of Medicine 2000;342(25):1878‐86. - PubMed
Beynon 2008 {published data only}
    1. Beynon R, Harris R, Sterne JAC, et al. The quantification of bias in randomised and non‐randomised studies: the BRANDO NRS Study [Poster]. 16th Cochrane Colloquium. Freiburg im Breisgau, Germany, 3‐7 October, 2008.
Bhandari 2004 {published data only}
    1. Bhandari M, Tornetta PIII, Ellis T, Audige L, Sprague S, Kuo JC, et al. Hierarchy of evidence: differences in results between non‐randomized studies and randomized trials in patients with femoral neck fractures. Archives of Orthopaedic and Trauma Surgery 2004;124(1):10‐6. - PubMed
Concato 2000 {published data only}
    1. Concato J, Shah N, Horwitz R. Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine 2000;342(25):1887‐92. - PMC - PubMed
Edwards 2012 {published data only}
    1. Edwards J, Kelly E, Lin Y, Lenders T, Ghali W, Graham A. Meta‐analytic comparison of randomized and nonrandomized studies of breast cancer surgery. Canadian Journal of Surgery 2012;55(3):155‐62. - PMC - PubMed
Furlan 2008 {published data only}
    1. Furlan A, Tomlinson G, Jadad A, Bombardier C. Examining heterogeneity in meta‐analysis: comparing results of randomized trials and nonrandomized studies of interventions for low back pain. Spine 2008;33(3):339‐48. - PubMed
Golder 2011 {published data only}
    1. Golder S, Loke YK, Bland M. Meta‐analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological review. PLoS Medicine 2011;8(5):e1001026. - PMC - PubMed
Ioannidis 2001 {published data only}
    1. Ioannidis J, Haidich A, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA 2001;286(7):821‐30. - PubMed
Kuss 2011 {published data only}
    1. Kuss O, Legler T, Boergermann J. Treatments effects from randomized trials and propensity score analyses were similar in similar populations in an example from cardiac surgery. Journal of Clinical Epidemiology 2011;64:1076‐84. - PubMed
Lonjon 2013 {published data only}
    1. Lonjon G, Boutron I, Trinquart L, Ahmad N, Aim F, Nizard R, et al. Comparison of treatment effect estimates from prospective nonrandomized studies with propensity score analysis and randomized controlled trials of surgical procedures. Annals of Surgery 2013;259(1):18‐25. - PubMed
Müeller 2010 {published data only}
    1. Müeller D, Sauerland S, Neugebauer EA, Immenroth M. Reported effects in randomized controlled trials were compared with those of nonrandomized trials in cholecystectomy. Journal of Clinical Epidemiology 2010;63(10):1082‐90. - PubMed
Naudet 2011 {published data only}
    1. Naudet F, Maria AS, Falissard B. Antidepressant response in major depressive disorder: a meta‐regression comparison of randomized controlled trials and observational studies. PLoS One 2011;6(6):e20811. - PMC - PubMed
Oliver 2010 {published data only}
    1. Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, et al. Randomised controlled trials for policy interventions: a review of reviews and meta‐regression. Health Technology Assessment 2010;14(16):1. - PubMed
Papanikolauo 2006 {published data only}
    1. Papanikolaou P, Christidi G, Ioannidis J. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ: Canadian Medical Association Journal 2006;174(5):635‐41. - PMC - PubMed
Shikata 2006 {published data only}
    1. Shikata S, Nakayama T, Noguchi Y, Taji Y, Yamagishi H. Comparison of effects in randomized controlled trials with observational studies in digestive surgery. Annals of Surgery 2006;244(5):668‐76. - PMC - PubMed

References to studies excluded from this review

Ather 2011 {published data only}
    1. Ather S, Bangalore S, Vemuri S, Cao LB, Bozkurt B, Messerli FH. Trials on the effect of cardiac resynchronization on arterial blood pressure in patients with heart failure. American Journal of Cardiology 2011;107(4):561‐78. - PubMed
Begg 1991 {published data only}
    1. Begg C, Pilote L. A model for incorporating historical controls into a meta‐analysis. Biometrics 1991;47(3):899‐906. - PubMed
Beyersmann 2008 {published data only}
    1. Beyersmann J, Gastmeier P, Wolkewitz M, Schumacher M. An easy mathematical proof showed that time‐dependent bias inevitably leads to biased effect estimation. Journal of Clinical Epidemiology 2008;61(12):1216‐21. - PubMed
Bosco 2010 {published data only}
    1. Bosco J, Silliman R, Thwin S, Geiger AM, Buist DS, Prout MN, et al. A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies. Journal of Clinical Epidemiology 2010;3(1):64‐74. - PMC - PubMed
Britton 1998 {published data only}
    1. Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Choosing between randomised and non‐randomised studies: a systematic review. Health Technology Assessment 1998;2(13):1‐124. - PubMed
Chambers 2010 {published data only}
    1. Chambers D, Fayter D, Paton F, Woolacott N. Use of non‐randomised evidence alongside randomised trials in a systematic review of endovascular aneurysm repair: strengths and limitations. European Journal of Vascular and Endovascular Surgery 2010;39(1):26‐34. - PubMed
Coulam 1994 {published data only}
    1. Coulam CB, Clark DA, Collins J, Scott JR, Schlesselman JS, Aoki K, et al. Recurrent Miscarriage Immunotherapy Trialists Group. Worldwide collaborative observational study and meta‐analysis on allogenic leukocyte immunotherapy for recurrent spontaneous abortion. American Journal of Reproductive Immunology 1994;32(2):55‐72. - PubMed
Dahabreh 2012 {published data only}
    1. Dahabreh I, Sheldrick R, Paulus J, Chung M, Varvarigou V, Jafri H, et al. Do observational studies using propensity score methods agree with randomized trials? A systematic comparison of studies on acute coronary syndromes. European Heart Journal 2012;33:1893‐901. - PMC - PubMed
Deeks 2002 {published data only}
    1. Deeks JJ, D'Amico R, Sakarovitch C, et al. Are comparability of case‐mix and the use of statistical adjustment markers of quality in non‐randomised studies? An empirical investigation. 4th Symposium on Systematic Reviews: Pushing the Boundaries. Oxford, UK, 2002.
Deeks 2003 {published data only}
    1. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. International Stroke Trial Collaborative Group, European Carotid Surgery Trial Collaborative Group. Evaluating non‐randomised intervention studies. Health Technology Assessment 2003;7(27):1‐173. - PubMed
Diehl 1986 {published data only}
    1. Diehl L, Perry D. A comparison of randomized concurrent control groups with matched historical control groups: are historical controls valid?. Journal of Clinical Oncology 1986;4(7):1114‐20. - PubMed
Diez 2010 {published data only}
    1. Diez P, Vogelius IS, Bentzen SM. A new method for synthesizing radiation dose‐response data from multiple trials applied to prostate cancer:. International Journal of Radiation Oncology, Biology, Physics 2010;77(4):1066‐71. - PMC - PubMed
Flossmann 2007 {published data only}
    1. Flossmann E, Rothwell P. Effect of aspirin on long‐term risk of colorectal cancer: consistent evidence from randomised and observational studies. Lancet 2007;369:1603‐13. - PubMed
Hallstrom 2000 {published data only}
    1. Hallstrom A, Anderson Jl, Cobb La, Friedman PL, Herre JM, Klein RC, et al. Advantages and disadvantages of trial designs: a review of analysis methods for ICD studies. Pacing and Clinical Electrophysiology: PACE 2000;23(6):1029‐38. - PubMed
Henry 2001 {published data only}
    1. Henry D, Moxey A, O'Connell D. Agreement between randomized and non‐randomized studies: the effects of bias and confounding. 9th Cochrane Colloquium. Lyon, France, 9‐13 October, 2001.
Hlatky 1988 {published data only}
    1. Hlatky MA, Califf RM, Harrell FE Jr, Lee KL, Mark DB, Pryor DB. Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery. Journal of the American College of Cardiology 1988;11(2):237‐45. - PubMed
Ioannidis 2005 {published data only}
    1. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005;294(2):218‐28. - PubMed
Labrarere 2006 {published data only}
    1. Labarere J, Bosson JL, Sevestre MA, Delmas AS, Dupas S, Thenault MH, et al. Graduated compression stocking thromboprophylaxis for elderly inpatients. Journal of General Internal Medicine 2006;21(12):1282‐7. - PMC - PubMed
LaTorre 2009 {published data only}
    1. LaTorre G, Waure C, Specchia ML, Nicolotti N, Capizzi S, Bilotta A, et al. Does quality of observational studies affect the results of a meta‐analysis?: the case of cigarette smoking and pancreatic cancer. Pancreas 2009;38(3):241‐7. - PubMed
Linde 2007 {published data only}
    1. Linde K, Streng A, Hoppe A, Weidenhammer W, Wagenpfeil S, Melchart D. Randomized trial vs. observational study of acupuncture for migraine found that patient characteristics differed but outcomes were similar. Journal of Clinical Epidemiology 2007;60(3):280‐7. - PubMed
Lipsey 1993 {published data only}
    1. Lipsey M, Wilson D. The efficacy of psychological, educational, and behavioral treatment. American Psychologist 1993;48(12):1181‐209. - PubMed
Loke 2011 {published data only}
    1. Loke Y, Cavallazzi R, Singh S. Risk of fractures with inhaled corticosteroids in COPD: systematic review and meta‐analysis of randomised controlled trials and observational studies. Thorax 2011;66:699‐708. - PubMed
MacLehose 2000 {published data only}
    1. MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AM. A systematic review of comparisons of effect sizes derived from randomised and non‐randomised studies. Health Technology Assessment 2000;4(34):1‐154. - PubMed
Mak 2009 {published data only}
    1. Mak A, Cheung MW, Chun‐Man Ho R, Ai‐Cia Cheak A, Chak Sing Lau C. Bisphosphonates and atrial fibrillation: Bayesian meta‐analyses of randomized controlled trials and observational studies. BMC Musculoskeletal Disorders 2009;10:113. - PMC - PubMed
McCarron 2010 {published data only}
    1. McCarron CE, Pullenayegum EM, Thabane L, Goeree R, Tarride JE. The importance of adjusting for potential confounders in Bayesian hierarchical models synthesising evidence from randomised and non‐randomised studies: an application comparing treatments for abdominalaortic aneurysms. BMC Medical Research Methodology 2010;10:64. - PMC - PubMed
McKee 1999 {published data only}
    1. McKee M, Britton A, Black N, McPherson K, Sanderson C, Bain C. Interpreting the evidence: choosing between randomised and non‐randomised studies. BMJ 1999;319:312‐5. - PMC - PubMed
Moreira 2012 {published data only}
    1. Moreira RF, Foltran FA, Albuquerque‐Sendin F, Mancini MC, Coury HJCG. Comparison of randomized and non‐ randomized controlled trials evidence regarding the effectiveness of workplace exercise on musculoskeletal pain control. Work 2012;41(Suppl 1):4782‐9. - PubMed
Ni Chroinin 2013 {published data only}
    1. Ni Chroinin D, Asplund K, Asberg S, Callaly E, Cuadrado‐Godia E, Diez‐Tejedor E, et al. Statin therapy and outcome after ischemic stroke: systematic review and meta‐analysis of observational studies and randomized trials. Stroke 2013;44(2):448‐56. - PubMed
Nixdorf 2010 {published data only}
    1. Nixdorf D, Moana‐Filho E, Law A, McGuire LA, Hodges JS, John MT. Frequency of persistent tooth pain after root canal therapy: a systematic review and meta‐analysis. Journal of Endodotics 2010;36(2):224‐30. - PMC - PubMed
Ottenbacker 1992 {published data only}
    1. Ottenbacher K. Impact of random assignment on study outcome: an empirical examination. Controlled Clinical Trials 1992;13:50‐61. - PubMed
Papanastassiou 2012 {published data only}
    1. Papanastassiou I, Phillips F, Meirhaeghe J, Berenson J, Andersson G, Chung G, et al. Comparing effects of kyphoplasty, vertebroplasty, and non‐ surgical management in a systematic review of randomized and non‐randomized controlled studies. European Spine Journal 2012;21:1826‐43. - PMC - PubMed
Phillips 1999 {published data only}
    1. Phillips AN, Grabar S, Tassie JM, Costagliola D, Lundgren JD, Egger M. Use of observational databases to evaluate the effectiveness of antiretroviral therapy for HIV infection: comparison of cohort studies with randomized trials. AIDS 1999;13(15):2075‐82. - PubMed
Pratt 2012 {published data only}
    1. Pratt N, Roughead E, Salter A, Ryan P. Choice of observational study design impacts on measurement of antipsychotic risks in the elderly: a systematic review. BMC Medical Research Methodology 2012;12(72):1‐19. - PMC - PubMed
Pyorala 1995 {published data only}
    1. Pyorala S, Huttunen N, Uhari M. A review and meta‐analysis of hormonal treatment of cryptorchidism. Journal of Clinical Endocrinology and Metabolism 1995;80(9):2795‐9. - PubMed
Schmoor 2008 {published data only}
    1. Schmoor C, Caputo A, Schumacher M. Evidence from nonrandomized studies: a case study on the estimation of causal effects. American Journal of Epidemiology 2008;167(9):1120‐9. - PubMed
Scott 2007 {published data only}
    1. Scott P, Kingsley G, Smith C, Choy EH, Scott DL. Non‐steroidal anti‐inflammatory drugs and myocardial infarctions: comparative systematic review of evidence from observational studies and randomised controlled trials. Annals of the Rheumatic Diseases 2007;66(10):1296‐304. - PMC - PubMed
Shah 2005 {published data only}
    1. Shah B, Laupacis A, Hux J, Austin PC. Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. Journal of Clinical Epidemiology 2005;58(6):550‐9. - PubMed
Shepherd 2006 {published data only}
    1. Shepherd J, Bagnall A, Colquitt J. 'Sometimes similar, sometimes different': a systematic review of meta‐analyses of randomised and non‐randomised policy intervention studies. 14th Cochrane Colloquium,. Dublin, Ireland, 23‐26 October, 2006.
Steinberg 1994 {published data only}
    1. Steinberg K, Smith J, Thacker S, Stroup DF. Breast cancer risk and duration of estrogen use: the role of study design in meta‐analysis. Epidemiology 1994;5(4):415‐21. - PubMed
Stukel 2007 {published data only}
    1. Stukel T, Fisher E, Wennberg D, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA 2007;297(3):278‐85. - PMC - PubMed
Ward 1992 {published data only}
    1. Ward LC, Fielding J, Dunn J, Kelly KA. The selection of cases for randomised trials:a registry survey of concurrent trial and non‐trial patients. British Journal of Cancer 1992;66(5):943‐50. - PMC - PubMed
Watson 1994 {published data only}
    1. Watson A, Vandekerckhove P, Lilford R, Vail A, Brosens I, Hughes E. A meta‐analysis of the therapeutic role of oil soluble contrast media at hysterosalpingography: a surprising result?. Fertility and Sterility 1994;61(3):470‐7. - PubMed
Williams 1981 {published data only}
    1. Williams PT, Fortmann SP, Farquhar JW, Varady A, Mellen S. A comparison of statistical methods for evaluating risk factor changes in community‐based studies: an example from the Stanford Three‐Community Study. Journal of Chronic Diseases 1981;34(11):565‐71. - PubMed
Wilson 2001 {published data only}
    1. Wilson D, Lipsey M. The role of method in treatment effectiveness research: evidence from meta‐analysis. Psychological Methods 2001;6(4):413‐29. - PubMed

Additional references

Altman 2003
    1. Altman D, Bland J. Interaction revisited: the difference between two estimates. BMJ 2003;326:219. - PMC - PubMed
Dorsey 2010
    1. Dorsey ER, Roulet J, Thompson JP, Reminick JL, Thai A, White‐Stellato Z, et al. Funding of US Biomedical Research, 2003‐2008. JAMA 2010;303(2):137‐43. - PMC - PubMed
Higgins 2011
    1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons Ltd, 2011.
Institute of Medicine 2009
    1. Institute of Medicine. Initial National Priorities for Comparitive Effectiveness Research. Institute of Medicine, Washington DC 2009.
Kamerow 2011
    1. Kamerow D. PCORI: odd name, important job, potential trouble. BMJ 2011;342:d2635. - PubMed
Kunz 1998
    1. Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non‐randomised clinical trials. BMJ 1998;317(7167):1185‐90. - PMC - PubMed
Kunz 2002
    1. Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews 2002, Issue 4. [DOI: 10.1002/14651858.MR000012] - DOI - PMC - PubMed
Lundh 2012
    1. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews 2012, Issue 12. [DOI: 10.1002/14651858.MR000033.pub2] - DOI - PubMed
Montori 2004
    1. Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005 Jan 8;330(7482):68. - PMC - PubMed
Odgaard‐Jensen 2011
    1. Odgaard‐Jensen J, Timmer A, Kunz R, Akl EA, Schünemann H, Briel M, et al. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews 2011, Issue 4. [DOI: 10.1002/14651858.MR000012.pub3] - DOI - PMC - PubMed
PCORI 2012
    1. Patient Centered Outcomes Research Institute (PCORI). PCORI Methodology Standards. http://www.pcori.org/assets/PCORI‐Methodology‐Standards.pdf December 14, 2012.
Reeves 2013
    1. Reeves B, Higgins J, Ramsay C, Shea B, Tugwall P, Wells G. An introduction to methodological issues when including non‐randomised studies in systematic reviews on the effects of interventions. Research Synthesis Methods 2013;4:1‐11. - PubMed
Sacks 1982
    1. Sacks H, Chalmers T, Smith HJ. Randomized versus historical controls for clinical trials. American Jourrnal of Medicine 1982;72(2):233‐40. - PubMed
Sampson 2009
    1. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence‐based practice guideline for the peer review of electronic search strategies. Journal of Clinical Epidemiology 2009;62(9):944‐52. - PubMed

Publication types

MeSH terms