Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jan 22:384:e074821.
doi: 10.1136/bmj-2023-074821.

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Richard D Riley  1   2 , Kym I E Snell  3   2 , Lucinda Archer  3   2 , Joie Ensor  3   2 , Thomas P A Debray  4 , Ben van Calster  5   6 , Maarten van Smeden  4 , Gary S Collins  7
Affiliations

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Richard D Riley et al. BMJ. .

Abstract

An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: support from EPSRC, NIHR-MRC, NIHR Birmingham Biomedical Research Centre, Cancer Research UK, FWO, and Internal Funds KU Leuven for the submitted work; RDR and GSC are statistical editors for The BMJ; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Figures

Fig 1
Fig 1
Illustration of the concern of low sample sizes when assessing calibration. Plot shows large variability in calibration curves from 100 external validation studies (each containing a random sample of 100 participants, and on average 43 outcome events, with outcomes generated assuming that the prediction model is truly well calibrated) of a prediction model for in-hospital clinical deterioration among admitted adults with covid-19
Fig 2
Fig 2
Summary of calculations for different sample sizes for external validation of a clinical prediction model for a continuous outcome (modified from Archer et al14), which target narrow confidence interval widths (as defined by 2×1.96×standard error) for key performance measures
Fig 3
Fig 3
Summary of different sample size criteria for external validation of a clinical prediction model for a binary outcome, as originally proposed by Riley et al. In criterion 3, the formula for the standard error of the c statistic was proposed by Newcombe. In criterion 4, the equation applied is derived by Marsh et al
Fig 4
Fig 4
Comparison of histogram (grey bars) of predicted values (estimated event probabilities) in the validation population of Gupta et al with our assumed beta distribution (curved line) used
Fig 5
Fig 5
Distribution of calibration curves for (A) pain intensity prediction model (based on 258 participants) and (B) covid-19 deterioration prediction model (based on 949 participants (408 events)), derived from 100 simulated datasets with the sample size required to estimate the calibration slope precisely according to a target confidence interval width of 0.3 (standard error ≤0.0765) for the calibration slope. Simulations assume that the models are well calibrated, with a true calibration slope of 1 and calibration-in-the-large of zero
See this image and copyright information in PMC

References

    1. Harrell FE, Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. 2nd ed. Springer, 2015. 10.1007/978-3-319-19425-7. - DOI
    1. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. 2nd ed. Springer, 2019. 10.1007/978-3-030-16399-0. - DOI
    1. Riley RD, van der Windt D, Croft P, Moons KGM, eds. Prognosis Research in Healthcare: Concepts, Methods and Impact. Oxford University Press, 2019. 10.1093/med/9780198796619.001.0001. - DOI
    1. Riley RD, Archer L, Snell KIE, et al. . Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ 2023;383:e074820. 10.1136/bmj-2023-074820 - DOI - PMC - PubMed
    1. Collins GS, de Groot JA, Dutton S, et al. . External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014;14:40. 10.1186/1471-2288-14-40 - DOI - PMC - PubMed

LinkOut - more resources