Prevalence and implications of significance testing for baseline covariate imbalance in randomised cancer clinical trials: The Table 1 Fallacy

Alexander D Sherry¹, Pavlos Msaouel², Zachary R McCaw³, Joseph Abi Jaoude⁴, Eric J Hsu⁵, Ramez Kouzy¹, Roshal Patel⁶, Yumeng Yang⁷, Timothy A Lin⁸, Cullen M Taniguchi⁹, Claus Rödel¹⁰, Emmanouil Fokas¹⁰, Chad Tang¹¹, Clifton David Fuller¹, Bruce Minsky¹², Tomer Meirson¹³, Ryan Sun¹⁴, Ethan B Ludmir¹⁵

Affiliations

¹ Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
² Department of Genitourinary Medical Oncology, Division of Cancer Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Translational Molecular Pathology, Division of Pathology/Lab Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
³ Insitro, South San Francisco, CA, USA; Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
⁴ Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Radiation Oncology, Stanford University, Stanford, CA, USA.
⁵ Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA.
⁶ Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Radiation Oncology, Memorial Sloan-Kettering Cancer Center, New York, NY, USA.
⁷ Department of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA.
⁸ Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
⁹ Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Experimental Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹⁰ Department of Radiotherapy and Oncology, University of Frankfurt, Frankfurt, Germany; Frankfurt Cancer Institute, Frankfurt, Germany; German Cancer Research Center (DKFZ), Heidelberg, German Cancer Consortium (DKTK), Partner Site Frankfurt am Main, Frankfurt, Germany.
¹¹ Department of Translational Molecular Pathology, Division of Pathology/Lab Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Genitourinary Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹² Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹³ Davidoff Cancer Center, Rabin Medical Center, Petach Tikva, Israel.
¹⁴ Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹⁵ Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. Electronic address: ebludmir@mdanderson.org.

PMID: 37827064
PMCID: PMC11512581
DOI: 10.1016/j.ejca.2023.113357

Prevalence and implications of significance testing for baseline covariate imbalance in randomised cancer clinical trials: The Table 1 Fallacy

Alexander D Sherry et al. Eur J Cancer. 2023 Nov.

. 2023 Nov:194:113357.

doi: 10.1016/j.ejca.2023.113357. Epub 2023 Sep 22.

Authors

Affiliations

¹ Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
² Department of Genitourinary Medical Oncology, Division of Cancer Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Translational Molecular Pathology, Division of Pathology/Lab Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
³ Insitro, South San Francisco, CA, USA; Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
⁴ Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Radiation Oncology, Stanford University, Stanford, CA, USA.
⁵ Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA.
⁶ Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Radiation Oncology, Memorial Sloan-Kettering Cancer Center, New York, NY, USA.
⁷ Department of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA.
⁸ Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
⁹ Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Experimental Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹⁰ Department of Radiotherapy and Oncology, University of Frankfurt, Frankfurt, Germany; Frankfurt Cancer Institute, Frankfurt, Germany; German Cancer Research Center (DKFZ), Heidelberg, German Cancer Consortium (DKTK), Partner Site Frankfurt am Main, Frankfurt, Germany.
¹¹ Department of Translational Molecular Pathology, Division of Pathology/Lab Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Genitourinary Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹² Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹³ Davidoff Cancer Center, Rabin Medical Center, Petach Tikva, Israel.
¹⁴ Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
¹⁵ Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. Electronic address: ebludmir@mdanderson.org.

PMID: 37827064
PMCID: PMC11512581
DOI: 10.1016/j.ejca.2023.113357

Abstract

Background: The 'Table 1 Fallacy' refers to the unsound use of significance testing for comparing the distributions of baseline variables between randomised groups to draw erroneous conclusions about balance or imbalance. We performed a cross-sectional study of the Table 1 Fallacy in phase III oncology trials.

Methods: From ClinicalTrials.gov, 1877 randomised trials were screened. Multivariable logistic regressions evaluated predictors of the Table 1 Fallacy.

Results: A total of 765 randomised controlled trials involving 553,405 patients were analysed. The Table 1 Fallacy was observed in 25% of trials (188 of 765), with 3% of comparisons deemed significant (59 of 2353), approximating the typical 5% type I error assertion probability. Application of trial-level multiplicity corrections reduced the rate of significant findings to 0.3% (six of 2345 tests). Factors associated with lower odds of the Table 1 Fallacy included industry sponsorship (adjusted odds ratio [aOR] 0.29, 95% confidence interval [CI] 0.18-0.47; multiplicity-corrected P < 0.0001), larger trial size (≥795 versus <280 patients; aOR 0.32, 95% CI 0.19-0.53; multiplicity-corrected P = 0.0008), and publication in a European versus American journal (aOR 0.06, 95% CI 0.03-0.13; multiplicity-corrected P < 0.0001).

Conclusions: This study highlights the persistence of the Table 1 Fallacy in contemporary oncology randomised controlled trials, with one of every four trials testing for baseline differences after randomisation. Significance testing is a suboptimal method for identifying unsound randomisation procedures and may encourage misleading inferences. Journal-level enforcement is a possible strategy to help mitigate this fallacy.

Keywords: Covariate imbalance; Oncology; Phase III; Randomised controlled trials; Significance testing for baseline characteristics; Table 1 Fallacy; Testing for baseline differences.

PubMed Disclaimer

Conflict of interest statement

Declaration of Competing Interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Dr. Msaouel reported honoraria for scientific advisory board membership for Mirati Therapeutics, Bristol-Myers Squibb, and Exelixis; consulting fees from Axiom Healthcare; non-branded educational programmes supported by Exelixis and Pfizer; leadership or fiduciary roles as a Medical Steering Committee Member for the Kidney Cancer Association and a Kidney Cancer Scientific Advisory Board Member for KCCure; and research funding from Takeda, Bristol-Myers Squibb, Mirati Therapeutics, and Gateway for Cancer Research (all unrelated to this manuscript’s content). Dr. McCaw reported employment at Insitro (unrelated to this manuscript’s content). Dr. Tang reported grants from the Cancer Prevention & Research Institute of Texas (CPRIT) and the Andrew Sabin Family Foundation and being an Andrew Sabin Scholar and royalties from Wolters Kluwer and consulting fees and honoraria from Bayer, Diffusion Pharmaceuticals, and The Osler Institute Lecture Series (unrelated to this manuscript’s content). Dr. Taniguchi reports consulting fees from Phebry and Xerient and holds a patent related to radioprotection of the upper gastrointestinal tract. Dr. Rödel reported receiving research funding from German Cancer Aid (unrelated to this manuscript’s content). Dr. Fuller reported unrelated funding and salary support from: NIH NIBIB Research Education Programmes for Residents and Clinical Fellows Grant (R25EB025787-01); NIDCR Academic Industrial Partnership Grant (R01DE028290); NCI Parent Research Project Grant (R01CA258827) an NIH/NCI Cancer Centre Support Grant (CCSG); and an NSF Division of Civil, Mechanical, and Manufacturing Innovation (CMMI) grant (NSF 1933369). Dr. Fuller reported direct industry grant support, honoraria, and travel funding from Elekta AB unrelated to this project and reported direct infrastructure support from the multidisciplinary Radiation Oncology/Cancer Imaging Programme (P30CA016672-44) of the MD Anderson Cancer Centre Support (Core) Grant (P30CA016672) and the MD Anderson Programme in Image-guided Cancer Therapy. Dr. Sun reported personal fees from Boehringer Ingelheim (unrelated to this manuscript’s content). Dr. Meirson reported receiving personal fees from Purple Biotech (unrelated to this manuscript’s content). No other authors reported any conflicts of interest.

Figures

**Fig. 1.**
The rate of significance testing for baseline variables over time according to publication year. The linear regression over time is shown with the shaded regions representing the 95% confidence interval of the slope (m). (A) All studied trials. (B) Trials published in journals defined as European. (C) Trials published in journals defined as American.

**Fig. 2.**
Structural causal model of the relationships between the journal locale, confounding variables, and the Table 1 Fallacy. A green rectangle indicates the exposure of interest (European versus American journal), and a blue rectangle represents the outcome of interest (Table 1 Fallacy). Yellow rectangles indicate confounders, and red rectangles indicate a non-confounding ancestor of both exposure and outcome. The green arrow represents the causal path, and the black arrows represent biasing paths.

See this image and copyright information in PMC

References

1. Altman DG. Comparability of randomised groups. J R Stat Soc Ser D (The Statistician) 1985;34(1):125–36.
1. Rothman KJ. Epidemiologic methods in clinical trials. Cancer 1977;39(4 Suppl):1771–5. - PubMed
1. Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Soc Sci Med 2018;210:2–21. - PMC - PubMed
1. Carter B Cluster size variability and imbalance in cluster randomized controlled trials. Stat Med 2010;29(29):2984–93. - PubMed
1. Senn S Seven myths of randomisation in clinical trials. Stat Med 2013;32(9):1439–50. - PubMed

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Medical
- MedlinePlus Health Information

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Prevalence and implications of significance testing for baseline covariate imbalance in randomised cancer clinical trials: The Table 1 Fallacy

Affiliations

Prevalence and implications of significance testing for baseline covariate imbalance in randomised cancer clinical trials: The Table 1 Fallacy

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Medical