Predicting Agitation Events in the Emergency Department Through Artificial Intelligence

Abstract

Importance: Agitation events are increasing in emergency departments (EDs), exacerbating safety risks for patients and clinicians. A wide range of clinical etiologies and behavioral patterns in the emergency setting make agitation prediction difficult in this setting.

Objective: To develop, train, and validate an agitation-specific prediction model based on a large, diverse set of past ED visit data.

Design, setting, and participants: This cohort study included electronic health record data collected from 9 ED sites within a large, urban health system in the Northeast US. All ED visits featuring patients aged 18 years or older from January 1, 2015, to December 31, 2022, were included in the analysis and modeling. Data analysis occurred between May 2023 and September 2024.

Exposures: Variables that served as potential exposures of interest, encompassing demographic information, patient history, initial vital signs, visit information, mode of arrival, and health services utilization.

Main outcomes and measures: The primary outcome of agitation was defined as the presence of an intramuscular chemical sedation and/or violent physical restraint order during an ED visit. A clinical model was developed to identify risk factors that predict agitation development during an ED visit prior to symptom onset. Model performance was measured using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (PR-AUC).

Results: The final cohort comprised 3 048 780 visits. The cohort had a mean (SD) age of 50.2 (20.4) years, with 54.7% visits among female patients. The final artificial intelligence model used 50 predictors for the primary outcome of predicting agitation events. The model achieved an AUROC of 0.94 (95% CI, 0.93-0.94) and a PR-AUC of 0.41 (95% CI, 0.40-0.42) in cross-validation, indicating good discriminative ability. Calibration of the model was evaluated and demonstrated robustness across the range of predicted probabilities. The top predictors in the final model included factors such as number of past ED visits, initial vital signs, medical history, chief concern, and number of previous sedation and restraint events.

Conclusions and relevance: Using a cross-sectional cohort of ED visits across 9 hospitals, the prediction model included factors for detecting risk of agitation that demonstrated high accuracy and applicability across diverse patient populations. These results suggest that clinical application of the model may enhance patient-centered care through preemptive deescalation and prevention of agitation.

Figure 1.. Flow Diagram for Emergency Department (ED) Visits Included in the Study and Breakdown of Visits Used for Training, Testing, and External Validation

Figure 2.. Model Performance and Evaluation Plots for the External Cohort

AUROC indicates area under the receiver operating characteristic curve; PR-AUC, area under the precision recall curve.

Figure 3.. Shapley Additive Explanations (SHAP) Bee Swarm Plot of the Top 25 Predictors of Agitation Events Generated From the Gradient-Boosted Decision Tree Model

Each dot corresponds to a single patient observation, and its position along the x-axis indicates how strongly and in which direction the predictor shifted the log-odds of an agitation event. Positive SHAP values (right of 0) indicate that a feature increased the log-odds (and thus the likelihood) of agitation, whereas negative SHAP values (left of 0) show a decrease in the log-odds. ED indicates emergency department; ESI, Emergency Severity Index.