Effective sample size for individual risk predictions: quantifying uncertainty in machine learning models

Abstract

Individual prediction uncertainty is a key aspect of clinical prediction model performance; however, standard performance metrics do not capture it. Consequently, a model might offer sufficient certainty for some patients but not for others, raising concerns about fairness. To address this limitation, the effective sample size has been proposed as a measure of sampling uncertainty. We developed a computational method to estimate effective sample sizes for a wide range of prediction models, including machine learning approaches. In this Viewpoint, we illustrated the approach using a clinical dataset (N=23 034) across five model types: logistic regression, elastic net, XGBoost, neural network, and random forest. During simulations, our approach generated accurate estimates of effective sample sizes for logistic regression and elastic net models, with minor deviations noted for the other three models. Although model performance metrics were similar across models, substantial differences in effective sample sizes and risk predictions were observed among patients in the clinical dataset. In conclusion, prediction uncertainty at the individual prediction level can be substantial even when models are developed using large samples. Effective sample size is thus a promising measure to communicate the uncertainty of predicted risk to individual users of machine learning-based prediction models.