Emergency department triage prediction of clinical outcomes using machine learning models

Abstract

Background: Development of emergency department (ED) triage systems that accurately differentiate and prioritize critically ill from stable patients remains challenging. We used machine learning models to predict clinical outcomes, and then compared their performance with that of a conventional approach-the Emergency Severity Index (ESI).

Methods: Using National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data, from 2007 through 2015, we identified all adult patients (aged ≥ 18 years). In the randomly sampled training set (70%), using routinely available triage data as predictors (e.g., demographics, triage vital signs, chief complaints, comorbidities), we developed four machine learning models: Lasso regression, random forest, gradient boosted decision tree, and deep neural network. As the reference model, we constructed a logistic regression model using the five-level ESI data. The clinical outcomes were critical care (admission to intensive care unit or in-hospital death) and hospitalization (direct hospital admission or transfer). In the test set (the remaining 30%), we measured the predictive performance, including area under the receiver-operating-characteristics curve (AUC) and net benefit (decision curves) for each model.

Results: Of 135,470 eligible ED visits, 2.1% had critical care outcome and 16.2% had hospitalization outcome. In the critical care outcome prediction, all four machine learning models outperformed the reference model (e.g., AUC, 0.86 [95%CI 0.85-0.87] in the deep neural network vs 0.74 [95%CI 0.72-0.75] in the reference model), with less under-triaged patients in ESI triage levels 3 to 5 (urgent to non-urgent). Likewise, in the hospitalization outcome prediction, all machine learning models outperformed the reference model (e.g., AUC, 0.82 [95%CI 0.82-0.83] in the deep neural network vs 0.69 [95%CI 0.68-0.69] in the reference model) with less over-triages in ESI triage levels 1 to 3 (immediate to urgent). In the decision curve analysis, all machine learning models consistently achieved a greater net benefit-a larger number of appropriate triages considering a trade-off with over-triages-across the range of clinical thresholds.

Conclusions: Compared to the conventional approach, the machine learning models demonstrated a superior performance to predict critical care and hospitalization outcomes. The application of modern machine learning models may enhance clinicians' triage decision making, thereby achieving better clinical care and optimal resource utilization.

Keywords: Critical care; Decision curve analysis; Emergency department; Hospital transfer; Hospitalization; Machine learning; Mortality; Prediction; Triage.

Fig. 1

Prediction ability of the reference model and machine learning models for intensive care use and in-hospital mortality in the test set. a Receiver-operating-characteristics (ROC) curves. The corresponding values of the area under the receiver-operating-characteristics curve (AUC) for each model are presented in Table 2. b Decision curve analysis. X-axis indicates the threshold probability for critical care outcome and Y-axis indicates the net benefit. Compared to the reference model, the net benefit for all machine learning models was larger over the range of clinical threshold

Fig. 2

Prediction ability of the reference model and machine learning models for hospitalization in the test set. a Receiver-operating-characteristics (ROC) curves. The corresponding values of the area under the receiver-operating-characteristics curve (AUC) for each model are presented in Table 2. b Decision curve analysis. X-axis indicates the threshold probability for hospitalization outcome and Y-axis indicates the net benefit. Compared to the reference model, the net benefit for all machine learning models was larger over the range of clinical threshold

Fig. 3

Variable importance of predictors in the random forest models. The variable importance is a scaled measure to have a maximum value of 100. The predictors with a variable importance of the top 15 are shown. a Critical care outcome. b Hospitalization outcome

Fig. 4

Variable importance of predictors in the gradient boosted decision tree models. The variable importance is a scaled measure to have a maximum value of 100. The predictors with a variable importance of top 15 are shown. a Critical care outcome. b Hospitalization outcome