Comparison of protocols and registry entries to published reports for randomised controlled trials
- PMID: 21249714
- PMCID: PMC7390503
- DOI: 10.1002/14651858.MR000031.pub2
Comparison of protocols and registry entries to published reports for randomised controlled trials
Abstract
Background: Publication of complete trial results is essential if people are to be able to make well-informed decisions about health care. Selective reporting of randomised controlled trials (RCTs) is a common problem.
Objectives: To systematically review studies of cohorts of RCTs to compare the content of trial reports with the information contained in their protocols, or entries in a trial registry.
Search strategy: We conducted electronic searches in Ovid MEDLINE (1950 to August 2010); Ovid EMBASE (1980 to August 2010); ISI Web of Science (1900 to August 2010) and the Cochrane Methodology Register (Issue 3, 2010), checked reference lists, and asked authors of eligible studies to identify further studies. Studies were not excluded based on language of publication or our assessment of their quality.
Selection criteria: Published or unpublished cohort studies comparing the content of protocols or trial registry entries with published trial reports.
Data collection and analysis: Data were extracted by two authors independently. Risk of bias in the cohort studies was assessed in relation to follow up and selective reporting of outcomes. Results are presented separately for the comparison of published reports to protocols and trial registry entries.
Main results: We included 16 studies assessing a median of 54 RCTs (range: 2 to 362). Twelve studies compared protocols to published reports and four compared trial registry entries to published reports. In two studies, eligibility criteria differed between the protocol and publication in 19% and 100% RCTs. In one study, 16% (9/58) of the reports included the same sample size calculation as the protocol. In one study, 6% (4/63) of protocol-report pairs gave conflicting information regarding the method of allocation concealment, and 67% (49/73) of blinded studies reported discrepant information on who was blinded. In one study unacknowledged discrepancies were found for methods of handling protocol deviations (44%; 19/43), missing data (80%; 39/49), primary outcome analyses (60%; 25/42) and adjusted analyses (82%; 23/28). One study found that of 13 protocols specifying subgroup analyses, 12 of these 13 trials reported only some, or none, of these. Two studies found that statistically significant outcomes had a higher odds of being fully reported compared to nonsignificant outcomes (range of odds ratios: 2.4 to 4.7). Across the studies, at least one primary outcome was changed, introduced, or omitted in 4-50% of trial reports.
Authors' conclusions: Discrepancies between protocols or trial registry entries and trial reports were common, although reasons for these were not discussed in the reports. Full transparency will be possible only when protocols are made publicly available or the quality and extent of information included in trial registries is improved, and trialists explain substantial changes in their reports.
Conflict of interest statement
Two of the authors of this review (PRW and DGA) are co‐authors of three studies included in the review (Hahn 2002; Chan 2004a; Chan 2004b).
Update of
- doi: 10.1002/14651858.MR000031
References
References to studies included in this review
Al‐Marzouki 2008 {published data only}
-
- Al‐Marzouki S, Roberts I, Evans S, Marshall T. Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet 2008;372(9634):201. - PubMed
Blumle 2008 {published data only}
-
- Blumle A, Antes G, Schumacher M, Just H, Elm E. Clinical research projects at a German medical faculty: follow‐up from ethical approval to publication and citation by others. Journal of Medical Ethics 2008;34:e20. - PubMed
-
- Blumle A, Meerpohl JJ, Antes G, Elm E. Reporting of eligibility criteria of participants in randomized trials: comparison between study protocols and journal articles [abstract]. 17th Cochrane Colloquium. Singapore, 2009:49‐50.
Bourgeois 2010 {published data only}
-
- Bourgeois FT, Murthy S, Mandl KD. Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov. Annals of Internal Medicine 2010;153:158‐166. - PMC - PubMed
Chan 2004a {published data only}
-
- Chan A‐W, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Journal of the American Medical Association 2004a;291:2457‐65. - PubMed
-
- Chan AW, Altman D. Discrepancies between protocols and publications: evidence of outcome reporting bias in randomised trials [abstract]. XI Cochrane Colloquium: Evidence, Health Care and Culture. Barcelona, Spain, 2003:9.
-
- Chan AW, Hrobjartsson A, Tendal B, Gotzsche P, Altman D. Pre‐specifying sample size calculations and statistical analyses in randomised trials: comparison of protocols to publications [abstract]. XIII Cochrane Colloquium. Melbourne, Australia, 2005:166.
-
- Gotzsche P, Hrobjartsson A, Hrobjartsonn A, Haahr M. Constraints on academic freedom in industry‐initiated clinical trials [abstract]. XIII Cochrane Colloquium. Melbourne, Australia, 2005:43.
Chan 2004b {published data only}
-
- Chan AW, Krleza‐Jeric K, Schmid I, Altman D. Selective reporting of results in government‐funded randomised trials [abstract]. 12th Cochrane Colloquium: Bridging the Gaps. Ottawa, Ontario, Canada, 2004:94‐95.
Charles 2009 {published data only}
Ewart 2009 {published data only}
Gandhi 2005 {published data only}
-
- Gandhi M, Ameli N, Bacchetti P, Sharp GB, French AL, Young M, Gange SJ, Anastos K, Holman S, Levine A, Greenblatt RM. Eligibility criteria for HIV clinical trials and generalizability of results: the gap between published reports and study protocols. AIDS 2005;19(16):1885‐96. - PubMed
Hahn 2002 {published data only}
-
- Hahn S, Williamson PR, Hutton JL. Investigation of within‐study selective reporting in clinical research: follow‐up of applications submitted to a local research ethics committee. Journal of Evaluations of Clinical Practice 2002;8:353‐9. - PubMed
Mathieu 2009 {published data only}
-
- Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. Journal of theAMA 2009;302(9):977‐84. - PubMed
Pich 2003 {published data only}
-
- Pich J, Carne X, Arnaiz JA, Gomez B, Trilla A, Rodes J. Role of a research ethics committee in follow‐up and publication of results. Lancet 2003;361:1015‐6. - PubMed
Scharf 2006 {published data only}
-
- Scharf O, Colevas AD. Adverse event reporting in publications compared with sponsor database for cancer clinical trials. Journal of Clinical Oncology 2006;24(24):3933‐3938. - PubMed
Shapiro 2000 {published data only}
-
- Shapiro SH, Weijer C, Freedman B. Reporting the study populations of clinical trials: Clear transmission or static on the line?. Journal of Clinical Epidemiology 2000;53:973‐979. - PubMed
Soares 2004 {published data only}
Vedula 2009 {published data only}
-
- Vedula SS, Bero L, Scherer RW, Dickersin K. Outcome reporting in industry‐sponsored trials of gabapentin for off‐label use. New England Journal of Medicine 2009;361(20):1963‐71. - PubMed
-
- Vedula SS, Dickersin K. Reporting of trials of gabapentin. New England Journal of Medicine 2010;362(17):1641‐2. - PubMed
von Elm 2008 {published data only}
-
- Elm E, Rollin A, Blumle A, Senessie C, Low N, Egger M. Selective reporting of outcomes of drug trials? Comparison of study protocols and published articles [abstract]. XIV Cochrane Colloquium. Dublin, Ireland, 2006:47.
-
- Elm E, Röllin A, Blümle A, Huwiler K, Witschi M, Egger M. Publication and non‐publication of clinical trials: longitudinal study of applications submitted to a research ethics committee. Swiss Medical Weekly 2008;138:197‐203. - PubMed
References to studies excluded from this review
Barcena 2005 {published data only}
-
- Barcena L, Pengel L, Morris PJ. Registry of randomized controlled trials in transplantation. Transplantation 2005;80(11):1525‐34. - PubMed
Bardy 1998 {published data only}
Berlin 2005 {published data only}
-
- Berlin JA, Wacholtz MC. Selective reporting, publication bias and clinical trial registry: an industry perspective. International Journal of Pharmaceutical Medicine 2005;19(5‐6):277‐284.
Chan 2005 {published data only}
Cooper 1997 {published data only}
-
- Cooper H, DeNeve K, Charlton K. Finding the missing science: the fate ofstudies submitted for review by a human subjects committee. Psychological Methods 1997;2(4):447‐452.
Cronin 2004 {published data only}
-
- Cronin E, Sheldon T. Factors influencing the publication of health research. International Journal of Technology Assessment in Health Care 2004;20:351‐5. - PubMed
Decullier 2005 {published data only}
Decullier 2006 {published data only}
Decullier 2007 {published data only}
Dickersin 1992 {published data only}
-
- Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow up of applications submitted to two institutional review boards. Journal of the American Medical Association 1992;267:374‐8. - PubMed
Dickersin 1993 {published data only}
-
- Dickersin K, Min YI. NIH clinical trials and publication bias. Online Journal of Current Clinical Trials 1993;Doc No. 50(53). - PubMed
Djulbegovic 2008 {published data only}
-
- Djulbegovic B, Kumar A, Soares H, Hozo I, Bepler G, Clarke M, Bennett C. New cancer treatment successes identified in phase 3 randomised controlled trials conducted by the national cancer institute‐sponsored cooperative oncology groups. Archives of Internal Medicine 2008;168(6):632‐642. - PMC - PubMed
Easterbrook 1991 {published data only}
-
- Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias inclinical research. Lancet 1991;337:867‐72. - PubMed
Habibzadeh 2006 {published data only}
Haidich 2001 {published data only}
-
- Haidich AB, Ioannidis JP. Effect of early patient enrolment on the time to completion and publication of randomized controlled trials. American Journal of Epidemiology 2001;154:873‐80. - PubMed
Hall 2007 {published data only}
-
- Hall R, Antueno C, Webber A. Publication bias in the medical literature: A review by a Canadian Research Ethics Board. Canadian Journal of Anesthesia 2007;54(5):380‐388. - PubMed
Ioannidis 1998 {published data only}
-
- Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. Journal of the American Medical Association 1998;279:281‐6. - PubMed
Lee 1998 {published data only}
-
- Lee HK, Lim KH, Park JH, Park KM, Kim HJ, Kim MY, Lee YS, Kim CJ, Chang JS, Shin SG. A survey of industrial perspectives on the Central Pharmaceutical Affairs Council's review of clinical trial protocols and study reports. Journal of Korean Society for Clinical Pharmacology & Therapeutics 1998;6(1):83‐100.
Liebeskind 1998 {published data only}
-
- Liebeskind DS, Kidwell CS, Saver JL. Empiric evidence of publication bias affecting acute stroke clinical trials. Stroke 1999;30(1):92. - PubMed
Liu 2008 {published data only}
-
- Liu XM, Li YP, Wu TX, Liu GJ, Li J. [A survey of the status of funding of registered Chinese trials]. Chinese Journal of Evidence‐based Medicine 2008;8(5):305‐311.
Melander 2003 {published data only}
Menzel 2007 {published data only}
-
- Menzel S, Uebing B, Hucklenbroich P, Schober O. Evaluation of clinicaltrials following an approval from a research ethics committee. Deutsche Medizinische Wochenschrift 2007;132(44):2313‐7. - PubMed
Nurbhai 2005 {published data only}
-
- Nurbhai M, Grimshaw J, Moja PL, Liberati A, Chan AW, Dickersin K, Krezla‐Jeric K, Moher D. Assessing the quality of information recorded on trial registries [abstract]. XIII Cochrane Colloquium. Melbourne, Australia, 2005:165.
Psaty 2008 {published data only}
-
- Psaty BM, Kronmal RA. Reporting mortality findings in trials of rofecoxib for alzheimer disease or cognitive impairment: a case study based on documents from rofecoxib litigation. Journal of the American Medical Association 2008;299(15):1813‐1817. - PubMed
Ramsey 2008 {published data only}
Rasmussen 2009 {published data only}
Rising 2008 {published data only}
Ross 2009 {published data only}
-
- Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.Gov: a cross‐sectional analysis. PLoS Medicine 2009;6(9):e1000144. - PMC - PubMed
Simes 1986 {published data only}
-
- Simes RJ. Publication bias: the case for an international registry of clinical trials. Journal of Clinical Oncology 1986;4(10):1529‐1541. - PubMed
Stern 1997 {published data only}
Turner 2008 {published data only}
-
- Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 2008;358(3):252‐60. - PubMed
References to studies awaiting assessment
Chappell 2005 {published data only}
-
- Chappell L, Alfirevich Z, Chien P, Jarvis S, Thornton JG. A comparison of the published version of randomized controlled trials in a specialist clinical journal with the original trial protocols [abstract]. International Congress on Peer Review and Biomedical Publication. Chicago, Illinois, USA, 2005:27.
Djulbegovic 2009 {published data only}
-
- Djulbegovic B, Kumar A, Magazin A, Soares HP. Quality of randomized controlled trials (RCTS) in hematological malignancies: What was reported versus what was done. Haematologica Conference: 14th Congress of the European Hematology Association. Berlin, Germany, 2009.
Djulbegovic 2010 {published data only}
Ghersi 2006 {unpublished data only}
-
- Ghersi D. Issues in the design, conduct and reporting of clinical trials that impact on the quality of decision making. School of Public Health, Faculty of Medicine. University of Sydney 2006.
-
- Ghersi D, Clarke M, Simes J. Selective reporting of the primary outcomes of clinical trials: a follow‐up study [abstract]. XIV Cochrane Colloquium. Dublin, Ireland, 2006.
Jureidini 2008 {published data only}
-
- Jureidini JN, McHenry LB, Mansfield PR. Clinical trials and drug promotion: Selective reporting of study 329. International Journal of Risk & Safety in Medicine 2008;20:73‐81.
Mhaskar 2009 {published data only}
-
- Mhaskar R, Kumar A, Soares H, Gardner B, Djulbegovic B. Treatment related harms: what was planned and what was reported? An analysis of Southwest Oncology Group phase III trials [abstract]. 17th Cochrane Colloquium. Singapore, 2009:49.
Smyth 2010 {published data only}
You 2010 {published data only}
-
- You B, Gan HK, Pond GR, Chen EX. Consistency in reporting of primary endpoints (PEP) from registration to publication for modern randomized oncology phase lll trials. American Society of Clinical Oncology. 2010:454s. - PubMed
References to ongoing studies
Chan 2010 {unpublished data only}
-
- personal communication.
McKenzie 2010 {unpublished data only}
-
- personal communication.
Rasmussen 2010 {unpublished data only}
-
- personal communication.
Urrutia 2010 {unpublished data only}
-
- personal communication.
Additional references
Chan 2004a
-
- Chan A‐W, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Journal of the American Medical Association 2004;291:2457‐65. - PubMed
Chan 2008a
Chan 2008b
-
- Chan A, Tetzlaff J, Altman DG, Gotzsche P, Hrobjartsson A, Krleza‐Jeric K, Laupacis A, Moher D. The SPIRIT initiative: defining standard protocol items for randomised trials. 16th Cochrane Colloquium, Freiburg, Germany (3‐7 October, 2008). German Journal for Evidence and Quality in Health Care. 2008; Vol. 102(Suppl VI):S27.
Chan 2008c
De Angelis 2004
-
- Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet 2004;364:911‐2. - PubMed
Dickersin 1987
-
- Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Controlled Clinical Trials 1987;8:343‐53. - PubMed
Dwan 2008
Ghersi 2009
-
- Ghersi D, Pang T. From Mexico to Mali: four years in thehistory of clinical trial registration. Journal of Evidence Based Medicine 2009;2(1):1‐7. - PubMed
Gøtzsche 2006
-
- Gøtzsche PC, Hrobjartsson A, Johansen KJ, Haahr MT, Altman DG, Chan AW. Constraints on Publication Rights in Industry‐Initiated Clinical Trials. Journal of the American Medical Association 2006;295(14):1645‐6. - PubMed
Gøtzsche 2007
Hahn 2000
-
- Hahn S, Williamson PR, Hutton JL, Garner P, Flynn EV. Assessing the potential for bias in meta‐analysis due to selective reporting of subgroup analyses within studies. Statistics in Medicine 2000;19:3325‐36. - PubMed
Hopewell 2009
Hrobjartsson 2009
-
- Hrobjartsson A, Pildal J, Chan AW, Haahr MT, Altman DG, Gøtzsche PC. Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. Journal of Clinical Epidemiology 2009;62(967):e973. - PubMed
Hutton 2000
-
- Hutton JL, Williamson PR. Bias in meta‐analysis due to outcome variable selection within studies. Applied Statistics 2000;49:359‐70.
Kirkham 2010
-
- Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth RMD, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. British Medical Journal 2010;340:c365. - PubMed
Moja 2009
Pildal 2005
Schulz 2010
Simes 1986
-
- Simes RJ. Publication bias: the case for an international registry of clinical trials. Journal of Clinical Oncology 1986;4:1529‐41. - PubMed
Tannock 1996
-
- Tannock IF. False‐positive results in clinical trials: multiple significance tests and the problem of unreported comparisons. Journal of the National Cancer Institute 1996;88:206‐7. - PubMed
Wang 2007
-
- Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine — Reporting of subgroup analyses in clinical trials. New England Journal of Medicine 2007;357(21):2189‐2194. - PubMed
WHO 2006
-
- World Health Organization. World Health Organization international clinical trials registry platform. New standards for registration of human medical research. http://www.who.int/mediacentre/news/releases/2006/pr25/en/ (Accessed 27 October 2010) 2006.
Williamson 2005a
-
- Williamson PR, Gamble C. Identification and impact of outcome selection bias in meta‐analysis. Statistics in Medicine 2005;24:1547‐61. - PubMed
Williamson 2005b
-
- Williamson PR, Gamble C, Altman DG, Hutton JL. Outcome selection bias in meta‐analysis. Statistical Methods in Medical Research 2005;14:515‐24. - PubMed
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources