Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Mar;35(3):1146-1156.
doi: 10.1007/s00330-024-11331-0. Epub 2025 Jan 9.

Overlooked and underpowered: a meta-research addressing sample size in radiomics prediction models for binary outcomes

Jingyu Zhong #  1   2   3 , Xianwei Liu #  4 , Junjie Lu #  5 , Jiarui Yang  6 , Guangcheng Zhang  7 , Shiqi Mao  8 , Haoda Chen  9 , Qian Yin  10 , Qingqing Cen  11 , Run Jiang  12 , Yang Song  13 , Minda Lu  14 , Jingshen Chu  15 , Yue Xing  4 , Yangfan Hu  4 , Defang Ding  4 , Xiang Ge  4 , Huan Zhang  16 , Weiwu Yao  17   18   19
Affiliations

Overlooked and underpowered: a meta-research addressing sample size in radiomics prediction models for binary outcomes

Jingyu Zhong et al. Eur Radiol. 2025 Mar.

Abstract

Objectives: To investigate how studies determine the sample size when developing radiomics prediction models for binary outcomes, and whether the sample size meets the estimates obtained by using established criteria.

Methods: We identified radiomics studies that were published from 01 January 2023 to 31 December 2023 in seven leading peer-reviewed radiological journals. We reviewed the sample size justification methods, and actual sample size used. We calculated and compared the actual sample size used to the estimates obtained by using three established criteria proposed by Riley et al. We investigated which characteristics factors were associated with the sufficient sample size that meets the estimates obtained by using established criteria proposed by Riley et al. RESULTS: We included 116 studies. Eleven out of one hundred sixteen studies justified the sample size, in which 6/11 performed a priori sample size calculation. The median (first and third quartile, Q1, Q3) of the total sample size is 223 (130, 463), and those of sample size for training are 150 (90, 288). The median (Q1, Q3) difference between total sample size and minimum sample size according to established criteria are -100 (-216, 183), and those differences between total sample size and a more restrictive approach based on established criteria are -268 (-427, -157). The presence of external testing and the specialty of the topic were associated with sufficient sample size.

Conclusion: Radiomics studies are often designed without sample size justification, whose sample size may be too small to avoid overfitting. Sample size justification is encouraged when developing a radiomics model.

Key points: Question Sample size justification is critical to help minimize overfitting in developing a radiomics model, but is overlooked and underpowered in radiomics research. Findings Few of the radiomics models justified, calculated, or reported their sample size, and most of them did not meet the recent formal sample size criteria. Clinical relevance Radiomics models are often designed without sample size justification. Consequently, many models are too small to avoid overfitting. It should be encouraged to justify, perform, and report the considerations on sample size when developing radiomics models.

Keywords: Methodology; Prediction model; Radiomics; Sample size.

PubMed Disclaimer

Conflict of interest statement

Compliance with ethical standards. Guarantor: The scientific guarantor of this publication is Prof. Weiwu Yao from the Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine. Conflict of interest: J.Z. acknowledges his position as a member of the Scientific Editorial Board of European Radiology, American Journal of Roentgenology, and BMC Medical Imaging. He has therefore not taken part in the review or selection process of this paper. R.J., from a commercial company, SciClone Pharmaceuticals (Holdings) Limited, is an expert in pharmacovigilance doing adverse drug reaction surveillance of anticancer drugs, without any payment and personal concern regarding this study. Y.S. and M.L., both from a commercial company, Siemens Healthineers Ltd., are an MR collaboration scientist and technicians doing technical support under Siemens collaboration regulation, without any payment and personal concern regarding this study. All other authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article. Statistics and biometry: One of the authors is an expert in statistics and biometry, who kindly provided statistical advice. Informed consent: Written informed consent was not required for this study because no human or animal subjects have been included in this study. Ethical approval: Ethical approval was not required for this study because no human or animal subjects have been included in this study. Study subjects or cohorts overlap: The abstract of this article entitled “Overlooked and underpowered: a meta-research study addressing sample size in radiomics research” (#10163) has been accepted as an oral presentation at the European Congress of Radiology 2025, February 26 - March 02, 2025, Vienna, Austria. The presenting author of this abstract is Dr. Jingyu Zhong. Methodology: Experimental Retrospective Performed at one institution

Figures

Fig. 1
Fig. 1
Flow diagram of study inclusion
Fig. 2
Fig. 2
Adherence of actual sample size to minimum sample size requirements. Scatter plots of (A) actual total sample size and minimum sample size according to criterion 3, and (B) actual sample size for training and minimum sample size according to all three criteria. The red, yellow, and blue dots indicate studies with actual total sample size did not meet the minimum sample size requirements according to Riley et al criterion 3, studies with actual total sample size met the minimum sample size requirements according to Riley et al criterion 3, and studies with actual sample size for training met the minimum sample size requirements according to Riley et al all three criteria, respectively. The dashed line indicates the actual total sample size equals the minimum sample size requirements. The upper left area indicates the study met the minimum sample size requirements; the bottom right area indicates the study did not meet the minimum sample size requirements
Fig. 3
Fig. 3
Comparison between the EPV ≥ 10 rule and Riley et al criteria. Scatter plot of a number of predictors in models against the actual number of events in model training. The red, yellow, and blue dots indicate studies with actual total sample size did not meet the minimum sample size requirements according to Riley et al criterion 3, studies with actual total sample size met the minimum sample size requirements according to Riley et al criterion 3, and studies with actual sample size for training met the minimum sample size requirements according to Riley et al all three criteria, respectively. The dashed line indicates the ten EPV rule. The upper left area indicates a study with EPV ≥ 10; the bottom right area indicates a study with EPV < 10
See this image and copyright information in PMC

Comment in

References

    1. Lambin P, Rios-Velazquez E, Leijenaar R et al (2012) Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 48:441–446. 10.1016/j.ejca.2011.11.036 - PMC - PubMed
    1. Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577. 10.1148/radiol.2015151169 - PMC - PubMed
    1. O’Connor JP, Aboagye EO, Adams JE et al (2017) Imaging biomarker roadmap for cancer studies. Nat Rev Clin Oncol 14:169–186. 10.1038/nrclinonc.2016.162 - PMC - PubMed
    1. Lambin P, Leijenaar RTH, Deist TM et al (2017) Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 14:749–762. 10.1038/nrclinonc.2017.141 - PubMed
    1. Huang EP, O’Connor JPB, McShane LM et al (2023) Criteria for the translation of radiomics into clinically useful tests. Nat Rev Clin Oncol 20:69–82. 10.1038/s41571-022-00707-0 - PMC - PubMed

LinkOut - more resources