Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2012 Jun;65(6):602-9.
doi: 10.1016/j.jclinepi.2011.10.016. Epub 2012 Mar 16.

Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols

Rahul Mhaskar  1 , Benjamin Djulbegovic, Anja Magazin, Heloisa P Soares, Ambuj Kumar
Affiliations

Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols

Rahul Mhaskar et al. J Clin Epidemiol. 2012 Jun.

Abstract

Objectives: To assess whether the reported methodological quality of randomized controlled trials (RCTs) reflects the actual methodological quality and to evaluate the association of effect size (ES) and sample size with methodological quality.

Study design and setting: Systematic review. This is a retrospective analysis of all consecutive phase III RCTs published by eight National Cancer Institute Cooperative Groups up to 2006. Data were extracted from protocols (actual quality) and publications (reported quality) for each study.

Results: Four hundred twenty-nine RCTs met the inclusion criteria. Overall reporting of methodological quality was poor and did not reflect the actual high methodological quality of RCTs. The results showed no association between sample size and actual methodological quality of a trial. Poor reporting of allocation concealment and blinding exaggerated the ES by 6% (ratio of hazard ratio [RHR]: 0.94; 95% confidence interval [CI]: 0.88, 0.99) and 24% (RHR: 1.24; 95% CI: 1.05, 1.43), respectively. However, actual quality assessment showed no association between ES and methodological quality.

Conclusion: The largest study to date shows that poor quality of reporting does not reflect the actual high methodological quality. Assessment of the impact of quality on the ES based on reported quality can produce misleading results.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest: All authors [RM, BD, AM, HS, AK] declare that they have no non-financial interests that may be relevant to the submitted work. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

Figures

Figure 1
Figure 1
Flow chart depicting the study selection process
Figure 2
Figure 2
Reporting of methodological quality domains associated with risk of bias in publication versus protocols.
Figure 3
Figure 3
Reporting of methodological quality domains associated with risk of random error in publication versus protocols.
Figure 4
Figure 4
Forest plot depicting ratio of hazard ratios for the outcome of overall survival (treatment effect) according to adequate versus inadequate reporting of methodological quality domains judged according to reporting in publications only versus protocol plus publications. Each rectangle represents the ratio of hazard ratios and the horizontal lines represent 95% confidence intervals (CI). A ratio of hazard ratios (RHR) < 1 implies that treatment effects were more beneficial in trials with adequate description of the methodological quality domain. A RHR of ≥1 implies that treatment effects were less beneficial in trials with adequate description of the methodological quality domain. A RHR of <1 along with associated CIs of <1 indicates a statistically significant association of methodological quality and treatment effect size.
See this image and copyright information in PMC

References

    1. Moher D, Ba P, Jones D, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP. Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analysis? Lancet. 1998;352:609–13. - PubMed
    1. Huwiler-Muntener KMD, Juni PMD, Junker CMDM, Egger MMDM. Quality of Reporting of Randomized Trials as a Measure of Methodologic Quality. JAMA PEER REVIEW CONGRESS IV. 2002;287(21):2801–4. http://www.jama.com. - PubMed
    1. Moher D, Jones A, Lepage L CONSORT Group (Consolidated Standards of Reporting Trials) Use of the CONSORT statement and quality of reports of randomized trials. A comparative before-and-after evaluation. JAMA. 2001;285:1992–5. - PubMed
    1. IOM. Finding what works in health care: statndards for systematic reviews. Washington DC: The national academics press; 2011. - PubMed
    1. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. Bmj. 2004 Jan 3;328(7430):22–4. - PMC - PubMed

Publication types