Machine Learning-Based Prediction of Clinical Outcomes for Children During Emergency Department Triage

Abstract

Importance: While machine learning approaches may enhance prediction ability, little is known about their utility in emergency department (ED) triage.

Objectives: To examine the performance of machine learning approaches to predict clinical outcomes and disposition in children in the ED and to compare their performance with conventional triage approaches.

Design, setting, and participants: Prognostic study of ED data from the National Hospital Ambulatory Medical Care Survey from January 1, 2007, through December 31, 2015. A nationally representative sample of 52 037 children aged 18 years or younger who presented to the ED were included. Data analysis was performed in August 2018.

Main outcomes and measures: The outcomes were critical care (admission to an intensive care unit and/or in-hospital death) and hospitalization (direct hospital admission or transfer). In the training set (70% random sample), using routinely available triage data as predictors (eg, demographic characteristics and vital signs), we derived 4 machine learning-based models: lasso regression, random forest, gradient-boosted decision tree, and deep neural network. In the test set (the remaining 30% of the sample), we measured the models' prediction performance by computing C statistics, prospective prediction results, and decision curves. These machine learning models were built for each outcome and compared with the reference model using the conventional triage classification information.

Results: Of 52 037 eligible ED visits by children (median [interquartile range] age, 6 [2-14] years; 24 929 [48.0%] female), 163 (0.3%) had the critical care outcome and 2352 (4.5%) had the hospitalization outcome. For the critical care prediction, all machine learning approaches had higher discriminative ability compared with the reference model, although the difference was not statistically significant (eg, C statistics of 0.85 [95% CI, 0.78-0.92] for the deep neural network vs 0.78 [95% CI, 0.71-0.85] for the reference; P = .16), and lower number of undertriaged critically ill children in the conventional triage levels 3 to 5 (urgent to nonurgent). For the hospitalization prediction, all machine learning approaches had significantly higher discrimination ability (eg, C statistic, 0.80 [95% CI, 0.78-0.81] for the deep neural network vs 0.73 [95% CI, 0.71-0.75] for the reference; P < .001) and fewer overtriaged children who did not require inpatient management in the conventional triage levels 1 to 3 (immediate to urgent). The decision curve analysis demonstrated a greater net benefit of machine learning models over ranges of clinical thresholds.

Conclusions and relevance: Machine learning-based triage had better discrimination ability to predict clinical outcomes and disposition, with reduction in undertriaging critically ill children and overtriaging children who are less ill.

Figure 1.. Prediction Ability of the Reference and Machine Learning Models for Intensive Care Unit Use and In-Hospital Mortality in the Test Set

A, Receiver operating characteristic curves. The corresponding values of the area under the curve for each model (ie, C statistics) are presented in Table 2. B, Decision curve analysis. The x-axis indicates the threshold probability for critical care outcome. The y-axis indicates the net benefit. The decision curves indicate the net benefit of models (the reference model and 4 machine learning models) as well as 2 clinical alternatives (classifying no children as having the outcome vs classifying all children as having the outcome) over a specified range of threshold probabilities of outcome. Compared with the reference model, the net benefit for all machine learning models was greater over the range of threshold probabilities.

Figure 2.. Prediction Ability of the Reference and Machine Learning Models for Hospitalization in the Test Set

A, Receiver operating characteristic curves. The corresponding values of the area under the curve for each model (ie, C statistics) are presented in Table 2. B, Decision curve analysis. The x-axis indicates the threshold probability for hospitalization outcome. The y-axis indicates the net benefit. The curves (decision curves) indicate the net benefit of models (the reference model and 4 machine learning models) as well as 2 clinical alternatives (classifying no children as having the outcome vs classifying all children as having the outcome) over a specified range of threshold probabilities of outcome. Compared with the reference model, the net benefit for all machine learning models was greater across the range of threshold probabilities, except the net benefit for the random forest model was lower for threshold probabilities below approximately 3%.

Figure 3.. Importance of Each Predictor in the Gradient-Boosted Decision Tree Models

The variable importance is a measure scaled to have a maximum value of 100. A, Critical care outcome. B, Hospitalization outcome. ED indicates emergency department.