Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Aug 25;15(8):e100411.
doi: 10.1136/bmjopen-2025-100411.

Disparity between statistical significance and clinical importance in published randomised controlled trials: a methodological study

Tonya Marianne Esterhuizen  1 , Lawrence Mbuagbaw  2   3   4   5 , Nadia Rehman  3 , Nathan Yanwou  6 , Devron J Swaby  7 , Esme Kittle  8 , Johann-Christoph Licht  8 , Lehana Thabane  2   3   4   5
Affiliations

Disparity between statistical significance and clinical importance in published randomised controlled trials: a methodological study

Tonya Marianne Esterhuizen et al. BMJ Open. .

Abstract

Objectives: We estimated the extent of the disparity between statistical significance and clinical importance in published randomised controlled trials (RCTs), and explored factors associated with this disparity.

Design: A methodological study of trials published between 2018 and 2022 and indexed in PubMed was conducted. Primary reports of two-arm, phase three, superiority trials of human health interventions were included. Pharmacokinetic studies and pilot trials were excluded. The relationship between the specified delta value or minimum clinically important difference (as specified in the sample size calculation) and the effect size determined the clinical importance of the trial results. Studies where the clinical importance was at least possible, with no statistical significance, were classified as SS-CI+ disparity, and studies which were definitely not clinically important but statistically significant were classified as SS+CI- disparity. Factors associated with each type of disparity were explored at the study level using multinomial logistic regression.

Results: 500 trials were included. In 38.4% (n=192) of these, information was not available to classify clinical importance. Overall disparity was found in 63 of the remaining 308 studies, 20.5% (95% confidence interval (CI) 16.2% to 25.5%). SS+CI- disparity was 10.3% (15/145) (95% CI 6.1% to 16.8%) and SS-CI+ disparity was 29.5% (48/163) (95% CI 22.7% to 37.2%).Studies testing complementary or alternative medicines relative to drug trials were positively associated with SS+CI- disparity. Low journal impact factor, small sample size, unfunded or grant funding and failure to mention allocation concealment were positively associated with SS-CI+disparity.

Conclusions: In up to 20% of RCTs, there may be a disparity between statistical significance and clinical importance. Clinical importance of results should be taken into account in the interpretation of trial results, and trials should adhere stringently to reporting guidelines.

Keywords: Clinical Relevance; Randomized Controlled Trial; Research Design; STATISTICS & RESEARCH METHODS.

PubMed Disclaimer

Conflict of interest statement

Competing interests: LT is a member of the Editorial Board of BMJ Open. The other authors declare no competing interests.

Figures

Figure 1
Figure 1
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram showing the number of studies identified and included from the database.
Figure 2
Figure 2
RCT quality characteristics of included studies (n=500). RCTs, randomised controlled trials.
See this image and copyright information in PMC

References

    1. García-Pérez MA. Thou Shalt Not Bear False Witness Against Null Hypothesis Significance Testing. Educ Psychol Meas 2017;77:631–62. 10.1177/0013164416668232 - DOI - PMC - PubMed
    1. Goodman S. A dirty dozen: twelve p-value misconceptions. Semin Hematol 2008;45:135–40. 10.1053/j.seminhematol.2008.04.003 - DOI - PubMed
    1. Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med 1999;130:1005–13. 10.7326/0003-4819-130-12-199906150-00019 - DOI - PubMed
    1. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat 2016;70:129–33. 10.1080/00031305.2016.1154108 - DOI
    1. Sterne JAC, Smith GD. Sifting the evidence-what’s wrong with significance tests. BMJ 2001;322:226–30. 10.1136/bmj.322.7280.226 - DOI - PMC - PubMed

MeSH terms

LinkOut - more resources