Machine Learning to Develop and Internally Validate a Predictive Model for Post-operative Delirium in a Prospective, Observational Clinical Cohort Study of Older Surgical Patients

Abstract

Background: Our objective was to assess the performance of machine learning methods to predict post-operative delirium using a prospective clinical cohort.

Methods: We analyzed data from an observational cohort study of 560 older adults (≥ 70 years) without dementia undergoing major elective non-cardiac surgery. Post-operative delirium was determined by the Confusion Assessment Method supplemented by a medical chart review (N = 134, 24%). Five machine learning algorithms and a standard stepwise logistic regression model were developed in a training sample (80% of participants) and evaluated in the remaining hold-out testing sample. We evaluated three overlapping feature sets, restricted to variables that are readily available or minimally burdensome to collect in clinical settings, including interview and medical record data. A large feature set included 71 potential predictors. A smaller set of 18 features was selected by an expert panel using a consensus process, and this smaller feature set was considered with and without a measure of pre-operative mental status.

Results: The area under the receiver operating characteristic curve (AUC) was higher in the large feature set conditions (range of AUC, 0.62-0.71 across algorithms) versus the selected feature set conditions (AUC range, 0.53-0.57). The restricted feature set with mental status had intermediate AUC values (range, 0.53-0.68). In the full feature set condition, algorithms such as gradient boosting, cross-validated logistic regression, and neural network (AUC = 0.71, 95% CI 0.58-0.83) were comparable with a model developed using traditional stepwise logistic regression (AUC = 0.69, 95% CI 0.57-0.82). Calibration for all models and feature sets was poor.

Conclusions: We developed machine learning prediction models for post-operative delirium that performed better than chance and are comparable with traditional stepwise logistic regression. Delirium proved to be a phenotype that was difficult to predict with appreciable accuracy.

Keywords: delirium; machine learning; model prediction; post-operative; statistical learning.

Fig. 1

Comparison of receiver operator curves (ROCs) for prediction of delirium by the various machine learning (ML) algorithms examined. a ROC curves when a measure of pre-operative cognitive function (3MS) was not included in the selected feature set; b ROC curves when 3MS was included in the selected feature set; c ROC curves for the full feature set.

Fig. 2

Violin plots showing the distribution of the probability of delirium across ML models and stepwise logistic regression for the full feature set. In addition to a marker for the median of the data and a box indicating the interquartile range (as in standard box plots), these violin plots also show the kernel probability density of the data at different values for non-delirious patients (green) and delirious patients (salmon). The horizontal bar indicates a detection prevalence of 25%.