Interpretable machine learning models for prolonged Emergency Department wait time prediction

Abstract

Objective: Prolonged Emergency Department (ED) wait times lead to diminished healthcare quality. Utilizing machine learning (ML) to predict patient wait times could aid in ED operational management. Our aim is to perform a comprehensive analysis of ML models for ED wait time prediction, identify key feature importance and associations with prolonged wait times, and interpret prediction model clinical relevance among ED patients.

Methods: This is a single-centered retrospective study. We included ED patients assigned an Emergency Severity Index (ESI) level of 3 at triage. Patient wait times were categorized as <30 minutes and ≥30 minutes (prolonged wait time). We employed five ML algorithms - cross-validation logistic regression (CVLR), random forest (RF), extreme gradient boosting (XGBoost), artificial neural network (ANN), and support vector machine (SVM) - for predicting patient prolonged wait times. Performance assessment utilized accuracy, recall, precision, F1 score, false positive rate (FPR), and false negative rate (FNR). Furthermore, using XGBoost as an example, model key features and partial dependency plots (PDP) of these key features were illustrated. Shapley additive explanations (SHAP) were employed to interpret model outputs. Additionally, a top key feature interaction analysis was conducted.

Results: Among total 177,665 patients, nearly half of them (48.20%, 85,632) experienced prolonged ED wait times. Though all five ML models exhibited similar performance, minimizing FNR is associated with the most clinical relevance for wait time predictions. The top features influencing patient wait times and gaining the top ranked interactions were ED crowding condition and patient mode of arrival.

Conclusions: Nearly half of the patients experienced prolonged wait times in the ED. ML models demonstrated acceptable performance, particularly in minimizing FNR when predicting ED wait times. The prediction of prolonged wait times was influenced by multiple interacting factors. Proper application of ML models to clinical practice requires interpreting their predictions of prolonged wait times in the context of clinical significance.

Keywords: Emergency department; Machine learning; Performance; Wait time.

Fig. 1

Comparison of false negative rates and false positive rates when different machine learning algorithms utilized for wait-time prediction. Figure 1 depicts various false negative rates (FNRs) and false positive rates (FPRs) for predicting patient wait times using different ML algorithms. Our focus was primarily on the FNR in our attempts to predict patients' wait times. A false negative occurs when ML algorithms misclassify a patient who waits longer than 30 minutes as waiting less than 30 minutes. In Figure 1, the highest FNR was observed when using RF algorithm to predict patient wait times, while the lowest FNR was observed when SVM algorithm was utilized. Abbreviations: FNR, False Negative Rate; CVLR, Cross Validation Logistic Regression; RF, Random Forest; XGBoost, eXtreme Gradient Boosting; ANN, Artificial Neural Network; SVM, Select Vector Machine

Fig. 2

Feature importance and associations from XGBoost classification model. Figure 2 illustrates the essential features contributing to wait time prediction using XGBoost algorithmic model. Panel A (Feature Importance): Feature importance of each feature contributing to the model prediction. The x-axis represents the marginal contribution of a feature to the change in the predicted probability of prolonged wait time (30min). Panel B (Feature Associations): The x-axis indicates the direction of each feature impact on the model output. SHAP values >0 indicates the prolonged wait time and <0 indicates patients wait time<30min. All features except age were dichotomous coded either 0 (no) or 1 (yes). For example, moa_ambulance (i.e., patients arrived by ambulance) had more negative values indicating the higher impact of predicting patient wait time<30min if patients arrived by ambulance

Fig. 3

Different SHAP values of the first 25 samples. Figure 3 shows the different SHAP values of the first 25 samples from this study. X-axis shows the number of samples and Y-axis shows the SHAP values of different features within the samples. It shows that certain features (i.e., red colored) can negatively contribute to the prolonged wait time predictions, whereas others (i.e., blue colored) may positively contribute to the wait time predictions. This figure shows the variability of each sample predicting prolonged ED wait times

Fig. 4

Partial dependency plots of leading predictors from XGBoost. Figure 4 depicts Partial Dependency Plots (PDP) generated using the XGBoost algorithmic model to predict patient wait time. The categorical features include mode of arrival (moa) by ambulance (Panel A) and ED crowding status (overly crowded, Panel B), while age is presented as a continuous feature (Panel C). The categorical features demonstrate bidirectional effects on patient wait time prediction. Generally, patients who arrived at the ED under not overly crowded conditions, as well as those who arrived by ambulance, experienced shorter wait times, whereas patients arriving at an overly crowded ED, or arrived not by ambulance, experienced longer wait times. The PDP for age reveals a complex pattern; patients at extreme ages (i.e., very young or very old) tended to experience shorter wait times compared to others

Fig. 5

Partial dependency plots of top feature interactions using XGBoost algorithm for prolonged wait time predictions. Figure 5 shows the PDP of top feature interactions. A shows the interactions between ED overly crowded conditions and ambulance transportation. When ED was overly crowded, more patients with ambulance transfer had less prolonged wait times than patients who arrived at ED without ambulance transfer. B shows when ED was not crowded, more patients who arrived without ambulance transfer had less prolonged wait times than patients who arrived via ambulance. C shows the interactions between age and ambulance transfer. Two perpendicular lines were drawn and indicated that patients’ age ranges (20 to 60). Generally, less prolonged wait time occurred when patients aged either younger than 20 or older than 60. ED wait times were quite similar among patients aged ranging from 20- to 60-year-old. When patients older than 60, more patients had less prolonged ED wait times when arrived at ED via ambulance than ones without