Advancing a machine learning-based decision support tool for pre-hospital assessment of dyspnoea by emergency medical service clinicians: a retrospective observational study

Abstract

Background: In Sweden with about 10 million inhabitants, there are about one million primary ambulance missions every year. Among them, around 10% are assessed by Emergency Medical Service (EMS) clinicians with the primary symptom of dyspnoea. The risk of death among these patients has been reported to be remarkably high, at 11,1% and 13,2%. The aim was to develop a Machine Learning (ML) model to provide support in assessing patients in pre-hospital settings and to compare them with established triage tools.

Methods: This was a retrospective observational study including 6,354 patients who called the Swedish emergency telephone number (112) between January and December 2017. Patients presenting with the main symptom of dyspnoea were included which were recruited from two EMS organisations in Göteborg and Södra Älvsborg. Serious Adverse Event (SAE) was used as outcome, defined as any of the following:1) death within 30 days after call for an ambulance, 2) a final diagnosis defined as time-sensitive, 3) admitted to intensive care unit, or 4) readmission within 72 h and admitted to hospital receiving a final time-sensitive diagnosis. Logistic regression, LASSO logistic regression and gradient boosting were compared to the Rapid Emergency Triage and Treatment System for Adults (RETTS-A) and National Early Warning Score2 (NEWS2) with respect to discrimination and calibration of predictions. Eighty percent (80%) of the data was used for model development and 20% for model validation.

Results: All ML models showed better performance than RETTS-A and NEWS2 with respect to all evaluated performance metrics. The gradient boosting algorithm had the overall best performance, with excellent calibration of the predictions, and consistently showed higher sensitivity to detect SAE than the other methods. The ROC AUC on test data increased from 0.73 (95% CI 0.70-0.76) with RETTS-A to 0.81 (95% CI 0.78-0.84) using gradient boosting.

Conclusions: Among 6,354 ambulance missions caused by patients suffering from dyspnoea, an ML method using gradient boosting demonstrated excellent performance for predicting SAE, with substantial improvement over the more established methods RETTS-A and NEWS2.

Keywords: Ambulance; Decision support tool; Dyspnoea; Emergency medical services; Machine learning; Prehospital; Serious adverse event.

Fig. 1

Flow chart of the studied patients, assessed with dyspnoea as the primary symptom

Fig. 2

Crude and age-adjusted relative risk of serious adverse event in univariable analyses. aRR, adjusted relative risk; CI, confidence interval; RR, relative risk

Fig. 3

Variable importance using Shapley values according to gradient boosting and logistic regression. Bars are normalized to sum to 100%

Fig. 4

ROC curves for prediction of serious adverse events using ML and traditional triage methods. A: Full curve. B: Zoomed in at sensitivity ≥ 95% and 1 − specificity ≥ 80%. ROC, receiver operating characteristic; TNR, true negative rate; TPR, true positive rate

Fig. 5

Calibration of gradient boosting for prediction of serious adverse events. A: Cross-validation performance on training data. B: Test data performance