Prediction of delirium occurrence using machine learning in acute stroke patients in intensive care unit

Abstract

Introduction: Delirium, frequently experienced by ischemic stroke patients, is one of the most common neuropsychiatric syndromes reported in the Intensive Care Unit (ICU). Stroke patients with delirium have a high mortality rate and lengthy hospitalization. For these reasons, early diagnosis of delirium in the ICU is critical for better patient prognosis. Therefore, we developed and validated prediction models to classify the real-time delirium status in patients admitted to the ICU or Stroke Unit (SU) with ischemic stroke.

Methods: A total of 84 delirium patients and 336 non-delirium patients in the ICU of Ajou University Hospital were included. The 8 fixed features [Age, Sex, Alcohol Intake, National Institute of Health Stroke Scale (NIHSS), HbA1c, Prothrombin time, D-dimer, and Hemoglobin] identified at admission and 12 dynamic features [Mean or Variability indexes calculated from Body Temperature (BT), Heart Rate (HR), Respiratory Rate (RR), Oxygen saturation (SpO2), Systolic Blood Pressure (SBP), and Diastolic Blood Pressure (DBP)] based on vital signs were used for developing prediction models using the ensemble method.

Results: The Area Under the Receiver Operating Characteristic curve (AUROC) for delirium-state classification was 0.80. In simulation-based evaluation, AUROC was 0.71, and the predicted probability increased closer to the time of delirium occurrence. We observed that the patterns of dynamic features, including BT, SpO2, RR, and Heart Rate Variability (HRV) kept changing as the time points were getting closer to the delirium occurrence time. Therefore, the model that employed these patterns showed increasing prediction performance.

Conclusion: Our model can predict the real-time possibility of delirium in patients with ischemic stroke and will be helpful to monitor high-risk patients.

Keywords: delirium; early diagnosis; ischemic stroke; machine learning; vital signs.

Figure 1

Overview of the study design. Fixed and dynamic features were used for the delirium prediction model development. For model development, dynamic features that were collected 2 h before the time point of delirium occurrence (case group) or matched time point (control group) were utilized. The stepwise method was used for feature selection. With the selected features, several machine learning algorithms were used and evaluated for model development. The final selected model was evaluated in a simulation environment by applying the model from 16 h before to the index time point every 2 h. W1–W8: the observation window 1–8; *the same dataset. BT, body temperature; HR, heart rate; RR, respiratory rate.

Figure 2

Input variable extraction strategy in each observation time window. (A) In addition to baseline data at admission, 2-h vital signs, the latest 30-min electrocardiogram, and body temperature measurement in each time window are used for model training. (B) All patients are matched by the length of stay in the intensive care unit using the propensity score matching method.

Figure 3

Inclusion and exclusion criteria for model development. All patients admitted to the intensive care unit (ICU) between July 2019 and December 2020 were enrolled. Two trained neurologists selected all patients by reviewing clinical data and nursing records. Finally, 67 patients with delirium (case group) and 268 patients without delirium (control group) were used for model development, and 90% were used for training and 10% for validation. In total, 17 patients with delirium and 68 patients without delirium were used for model evaluation. *AUMC, Ajou University Medical Center; CAM, confusion assessment method.

Figure 4

Dataset split workflow. The total dataset is divided by patient into the training set (80%) and test set (20%). Then, 10% of the training set is divided into the validation set for hyperparameter tuning of the model.

Figure 5

Delirium occurrence alarm system embedded in the prediction model. This image shows the clinical dashboard user interface used for temporal validation. (A) Alarm Dashboard page, (B) Model Prediction and vital sign trend view page, and (C) Data input page.

Figure 6

Feature selection results. This plot shows the final variable list selected through stepwise feature selection method. The final ML model included top 20 features.

Figure 7

Comparison of prediction performance across different Machine Learning algorithms. This plot shows the Receiver Operating Characteristic curves of each model that is trained by different algorithms (LR, RF, SVM, LGBM, XGB) using the top 20 variables. These results are calculated from the mean performances of 50 models trained by different randomly selected training datasets. The red area represents the mean ± SD of the performance of the final model.

Figure 8

Comparison of models’ performance according to the used features. Comparison of models’ performance according to the used features. The AUROC was higher (0.80) when fixed and dynamic features were used together, compared to the models that used only fixed (0.70) or dynamic features (0.71).

Figure 9

Mean prediction probability and mean number of alarms per patient in the retrospective validation. It shows the mean prediction probability of the model by time point (above), and the number of alarms per patient by time point (below). The orange line indicates delirium group and green line indicates non-delirium group. The black dashed line indicates the time point before 30 min of actual delirium event occurrence. The model’s prediction probability is higher and higher recall performance is seen closer to the event occurrence.

Figure 10

Mean prediction probability and mean number of alarms per patient in the temporal validation. The image shows the mean prediction probability of the model by time point (A) and the number of alarms per patient by time point (B). Similar to the retrospective validation, we saw a gradual increase in prediction probability and the number of alarms closer to the time of the event, with this pattern starting around 48 h before the delirium.