Dementia risk prediction in individuals with mild cognitive impairment: a comparison of Cox regression and machine learning models

Abstract

Background: Cox proportional hazards regression models and machine learning models are widely used for predicting the risk of dementia. Existing comparisons of these models have mostly been based on empirical datasets and have yielded mixed results. This study examines the accuracy of various machine learning and of the Cox regression models for predicting time-to-event outcomes using Monte Carlo simulation in people with mild cognitive impairment (MCI).

Methods: The predictive accuracy of nine time-to-event regression and machine learning models were investigated. These models include Cox regression, penalized Cox regression (with Ridge, LASSO, and elastic net penalties), survival trees, random survival forests, survival support vector machines, artificial neural networks, and extreme gradient boosting. Simulation data were generated using study design and data characteristics of a clinical registry and a large community-based registry of patients with MCI. The predictive performance of these models was evaluated based on three-fold cross-validation via Harrell's concordance index (c-index), integrated calibration index (ICI), and integrated brier score (IBS).

Results: Cox regression and machine learning model had comparable predictive accuracy across three different performance metrics and data-analytic conditions. The estimated c-index values for Cox regression, random survival forests, and extreme gradient boosting were 0.70, 0.69 and 0.70, respectively, when the data were generated from a Cox regression model in a large sample-size conditions. In contrast, the estimated c-index values for these models were 0.64, 0.64, and 0.65 when the data were generated from a random survival forest in a large sample size conditions. Both Cox regression and random survival forest had the lowest ICI values (0.12 for a large sample size and 0.18 for a small sample size) among all the investigated models regardless of sample size and data generating model.

Conclusion: Cox regression models have comparable, and sometimes better predictive performance, than more complex machine learning models. We recommend that the choice among these models should be guided by important considerations for research hypotheses, model interpretability, and type of data.

Keywords: Cox regression; Dementia; Machine learning; Risk prediction; Time-to-event outcomes.

Fig. 1

The process of simulation was conducted as follows:

Fig. 2

Distribution of the estimated c-index of nine models, assessed from three-fold CV across 500 replications NB: There are four panels, with the top two panels (A and B) are for the small samples (based on characteristics and features of PROMPT dataset), the bottom two panels (C and D) are for the large samples (based on characteristics and features of NACC dataset). Left and right panel are for the Cox regression used for data generating process [DGP] and random survival forests [RSF] based DGP, respectively. Each panel consists of nine boxplots corresponding to each of the nine survival analysis models. Each boxplot shows the variation in the Harrell’s c-index [c-index] across the 500 simulation replicates when a certain DGP and survival analysis method were applied. Cox: Cox proportional hazards; Ridge-Cox: Cox regression based on ridge penalty; LASSO-Cox: Cox regression based on Least Absolute Shrinkage Selection Operator penalty; EN-Cox: Cox regression based on elastic net penalty; SurvTree: Survival Tree; RSF: Random survival forests; SSVM: Survival support vector machine; SNN: Survival neural networks; XGBoost: Extreme gradient boosting

Fig. 3

Distribution of the estimated ICI of nine models, assessed from three-fold CV across 500 replications NB: There are four panels, with the top two panels (A and B) are for the small samples (based on characteristics and features of PROMPT dataset), the bottom two panels (C and D) are for the large samples (based on characteristics and features of NACC dataset). Left and right panel are for the Cox regression used for data generating process [DGP] and random survival forests [RSF] based DGP, respectively. Each panel consists of nine boxplots corresponding to each of the nine survival analysis models. Each boxplot shows the variation in the integrated calibration index [ICI] across the 500 simulation replicates when a certain DGP and survival analysis method were applied). Cox: Cox proportional hazards; Ridge-Cox: Cox regression based on ridge penalty; LASSO-Cox: Cox regression based on Least Absolute Shrinkage Selection Operator penalty; EN-Cox: Cox regression based on elastic net penalty; SurvTree: Survival Tree; RSF: Random survival forests; SSVM: Survival support vector machine; SNN: Survival neural networks; XGBoost: Extreme gradient boosting

Fig. 4

Distribution of the estimated IBS of nine models, assessed from three-fold CV across 500 replications NB: There are four panels, with the top two panels (A and B) are for the small samples (based on characteristics and features of PROMPT dataset), the bottom two panels (C and D) are for the large samples (based on characteristics and features of NACC dataset). Left and right panel are for the Cox regression used for data generating process [DGP] and random survival forests [RSF] based DGP, respectively. Each panel consists of nine boxplots corresponding to each of the nine survival analysis models. Each boxplot shows the variation in the integrated brier score [IBS] across the 500 simulation replicates when a certain DGP and survival analysis method were applied). Cox: Cox proportional hazards; Ridge-Cox: Cox regression based on ridge penalty; LASSO-Cox: Cox regression based on Least Absolute Shrinkage Selection Operator penalty; EN-Cox: Cox regression based on elastic net penalty; SurvTree: Survival Tree; RSF: Random survival forests; SSVM: Survival support vector machine; SNN: Survival neural networks; XGBoost: Extreme gradient boosting