Reproducing Statistics Performed in Published Randomized Controlled Trials

Abstract

Introduction: Randomized controlled trials (RCT) represent evidence at the lowest potential risk for bias. Clinicians in all specialties depend upon RCTs to guide patient care. Issues such as statistical discordance, or reporting statistical results that cannot be reproduced, should be uncommon. Our aim was to confirm the statistical reproducibility of published RCTs.

Methods: PubMed was searched using "randomized controlled trial." Studies were selected using a random number generator. Studies were included if the primary outcome could be reproduced using the data and statistical test reported in the manuscript. The reproduced p-value from our analysis and the published p-value were compared. Primary outcome was the number of studies that reported p-values that differed in statistical significance (crossed p-value=0.05) from the reproduction analysis. Assuming an alpha of 0.05, a beta of 0.80, an estimated rate of statistical discordance of 5% for RCTs, a total of at least 568 studies were required.

Results: Overall, 572 RCTs were selected involving six specialties. Of these, 45% were positive (p<0.05) studies. Eleven (2%) published results that differed from the reproduction analysis and crossed the p=0.05 threshold. All 11 studies were positive studies (while the reproduction analysis demonstrated p≥0.05).

Conclusion: Less than 5% of published RCTs reported a discordant p-value that crossed the "p=0.05" threshold. Although the occurrence is uncommon, the existence of even one RCT publishing nonreproducible results is concerning. Future studies should seek to identify why some RCTs report discordant statistics and how to prevent this from occurring.

Figure 1

Flowsheet of included studies.