Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2009 Jan 21;2009(1):MR000006.
doi: 10.1002/14651858.MR000006.pub3.

Publication bias in clinical trials due to statistical significance or direction of trial results

Sally Hopewell  1 , Kirsty Loudon, Mike J Clarke, Andrew D Oxman, Kay Dickersin
Affiliations

Publication bias in clinical trials due to statistical significance or direction of trial results

Sally Hopewell et al. Cochrane Database Syst Rev. .

Abstract

Editorial note: There is a more recent Cochrane review on this topic: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR000011.pub3/full.

Background: The tendency for authors to submit, and of journals to accept, manuscripts for publication based on the direction or strength of the study findings has been termed publication bias.

Objectives: To assess the extent to which publication of a cohort of clinical trials is influenced by the statistical significance, perceived importance, or direction of their results.

Search strategy: We searched the Cochrane Methodology Register (The Cochrane Library [Online] Issue 2, 2007), MEDLINE (1950 to March Week 2 2007), EMBASE (1980 to Week 11 2007) and Ovid MEDLINE In-Process & Other Non-Indexed Citations (March 21 2007). We also searched the Science Citation Index (April 2007), checked reference lists of relevant articles and contacted researchers to identify additional studies.

Selection criteria: Studies containing analyses of the association between publication and the statistical significance or direction of the results (trial findings), for a cohort of registered clinical trials.

Data collection and analysis: Two authors independently extracted data. We classified findings as either positive (defined as results classified by the investigators as statistically significant (P < 0.05), or perceived as striking or important, or showing a positive direction of effect) or negative (findings that were not statistically significant (P >/= 0.05), or perceived as unimportant, or showing a negative or null direction in effect). We extracted information on other potential risk factors for failure to publish, when these data were available.

Main results: Five studies were included. Trials with positive findings were more likely to be published than trials with negative or null findings (odds ratio 3.90; 95% confidence interval 2.68 to 5.68). This corresponds to a risk ratio of 1.78 (95% CI 1.58 to 1.95), assuming that 41% of negative trials are published (the median among the included studies, range = 11% to 85%). In absolute terms, this means that if 41% of negative trials are published, we would expect that 73% of positive trials would be published.Two studies assessed time to publication and showed that trials with positive findings tended to be published after four to five years compared to those with negative findings, which were published after six to eight years. Three studies found no statistically significant association between sample size and publication. One study found no significant association between either funding mechanism, investigator rank, or sex and publication.

Authors' conclusions: Trials with positive findings are published more often, and more quickly, than trials with negative findings.

PubMed Disclaimer

Conflict of interest statement

Kay Dickersin was the primary investigator of two of the included studies.

Figures

1
1
Forest plot of comparison: 1 Rate of publication and significance of trial result (pooled), outcome: 1.1 Total number of trials published.
2
2
Forest plot of comparison: 2 Rate of publication and significance of trial result (unpooled), outcome: 2.1 Total number of trials published.
1.1
1.1. Analysis
Comparison 1: Rate of publication and significance of trial result (pooled), Outcome 1: Total number of trials published
2.1
2.1. Analysis
Comparison 2: Rate of publication and significance of trial result (unpooled), Outcome 1: Total number of trials published
See this image and copyright information in PMC

Update of

References

References to studies included in this review

Bardy 1998 {published data only}
    1. Bardy AH. Bias in reporting clinical trials. British Journal of Clinical Pharmacology 1998;46:147-50. - PMC - PubMed
Dickersin 1992 {published and unpublished data}
    1. Dickersin K, Min Yi, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267:374-8. - PubMed
Dickersin 1993 {published data only}
    1. Dickersin K, Min Yi. NIH clinical trials and publication bias. Online Journal of Current Clinical Trials (serial online) 1993;(Doc-No 50). - PubMed
Ioannidis 1998 {published data only}
    1. Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281-6. - PubMed
Stern 1997 {published data only}
    1. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640-5. - PMC - PubMed

References to studies excluded from this review

Chan 2004A {published data only}
    1. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.. JAMA 2004;291:2457-65. - PubMed
    1. Hrobjartsson A, Chan AW, Haahr MT, Gotzsche PC, Altman DG. Selective reporting of positive outcomes in randomised trials - secondary publication. A comparison of protocols with published reports. Ugeskrift for Laeger 2005;167:3189-91. - PubMed
Chan 2004B {published data only}
    1. Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004;171:735-40. - PMC - PubMed
Cronin 2004 {published data only}
    1. Cronin E, Sheldon T. Factors influencing the publication of health research. International Journal of Technology Assessment in Health Care 2004;20:351-5. - PubMed
Decullier 2005 {published data only}
    1. Decullier E, Lheritier V, Chapuis F. Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ 2005;331:19-22. - PMC - PubMed
Easterbrook 1991 {published data only}
    1. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. The Lancet 1991;337:867-72. - PubMed
Hahn 2002 {published data only}
    1. Hahn S, Williamson PR, Hutton JL. Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. Journal of Evaluation in Clinical Practice 2002;8:353-9. - PubMed
Melander 2003 {published data only}
    1. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 2003;326:1171-3. - PMC - PubMed
Misakian 1998 {published data only}
    1. Misakian AL, Bero LA. Publication bias and research on passive smoking. JAMA 1998;280:250-3. - PubMed
Pich 2003 {published data only}
    1. Pich J, Carne X, Arnaiz JA, Gomez B, Trilla A, Rodes J. Role of a research ethics committee in follow-up and publication of results. The Lancet 2003;361:1015-6. - PubMed
Wormald 1997 {published data only}
    1. Wormald R, Bloom J, Evans J, Oldfield K. Publication bias in eye trials. In: 2nd International Conference Scientific Basis of Health Services & 5th Annual Cochrane Colloquium, 1997 5-8 October; Amsterdam, The Netherlands.

References to studies awaiting assessment

Menzel 2007 {published data only}
    1. Menzel S, Uebing B, Hucklenbroich P, Schober O. Evaluation of clinical trials following an approval from a research ethics committee. Deutsche Medizinische Wochenschrift 2007;132:2313-7. - PubMed
Turner 2008 {published data only}
    1. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. The New England Journal of Medicine 2008;358:252-60. - PubMed

Additional references

Bero 1996
    1. Bero LA, Rennie D. Influences on the quality of published drug studies. International Journal of Technology Assessment in Health Care 1996;12:209-37. - PubMed
Callaham 1998
    1. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA 1998;280:254-7. - PubMed
Chalmers 1990
    1. Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263:1405-8. - PubMed
De Angelis 2005
    1. De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. The Lancet 2005;365:1827-9. - PubMed
Dickersin 1987
    1. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. Publication bias and clinical trials. Controlled Clinical Trials 1987;8:343-53. - PubMed
Dickersin 1990
    1. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;206:1385-9. - PubMed
Dickersin 1997
    1. Dickersin K. How important is publication bias? A synthesis of available data. AIDS Education & Prevention 1997;9(Suppl A):15-21. - PubMed
Dickersin 2002
    1. Dickersin K, Olson CM, Rennie D, Cook D, Flanagin A, Zhu Q, et al. Association between time interval to publication and statistical significance. JAMA 2002;287:2829-31. - PubMed
Djulbegovic 2000
    1. Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, Adams JR, et al. The uncertainty principle and industry-sponsored research. The Lancet 2000;356:635-8. - PubMed
Donaldson 1996
    1. Donaldson IJ, Cresswell PA. Dissemination of the work of public health medicine trainees in peer-reviewed publications: an unfulfilled potential. Public Health 1996;110:61-3. - PubMed
Egger 1997
    1. Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. The Lancet 1997;350:326-9. - PubMed
Gulmezoglu 2005
    1. Gulmezoglu AM, Pang T, Horton R, Dickersin K. WHO facilitates international collaboration in setting standards for clinical trial registration. The Lancet 2005;365:1829-31. - PubMed
Hopewell 2007A
    1. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No: MR000010. [DOI: 10.1002/14651858.MR000010.pub3] - DOI - PMC - PubMed
Hopewell 2007B
    1. S Hopewell, M Clarke, L Stewart, J Tierney. Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No: MR000011. [DOI: 10.1002/14651858.MR000011.pub2] - DOI - PMC - PubMed
Jüni 2002
    1. Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International Journal of Epidemiology 2002;31:115-23. - PubMed
Kjaergard 2002
    1. Kjaergard LL, Als-Nielsen B KKjaergard LL, Als-Nielsen B. Association between competing interests and authors conclusions: epidemiological study of randomised clinical trials published in the BMJ. BMJ 2002;325:249. - PMC - PubMed
Lexchin 2003
    1. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. 2003 BMJ;326:1167-70. - PMC - PubMed
Olson 2002
    1. Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, et al. Publication bias in editorial decision making. JAMA 2002;287:2825-8. - PubMed
RevMan 5 [Computer program]
    1. Review Manager (RevMan). Version 5.0. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2008.
Scherer 2007
    1. Scherer RW, Langenberg P, Elm E. Full publication of results initially presented in abstracts. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No: MR000005. [DOI: 10.1002/14651858.MR000005.pub3] - DOI - PubMed
Simes 1986
    1. Simes RJ. Publication bias: the case for an international registry of clinical trials. Journal of Clinical Oncology 1986;4:1529-41. - PubMed
Simes 1987
    1. Simes RJ. Confronting publication bias: a cohort design for meta-analysis. Statistics in Medicine 1987;6:11-29. - PubMed
Song 2000
    1. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technology Assessment 2000;4(10). - PubMed
Tonks 2002
    1. Tonks A. A clinical trials register for Europe. BMJ 2002;325:1314-5. - PMC - PubMed
Weber 1998
    1. Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA 1998;280:257-9. - PubMed

Publication types

MeSH terms