Pilot deployment of a machine-learning enhanced prediction of need for hemorrhage resuscitation after trauma - the ShockMatrix pilot study

Abstract

Importance: Decision-making in trauma patients remains challenging and often results in deviation from guidelines. Machine-Learning (ML) enhanced decision-support could improve hemorrhage resuscitation.

Aim: To develop a ML enhanced decision support tool to predict Need for Hemorrhage Resuscitation (NHR) (part I) and test the collection of the predictor variables in real time in a smartphone app (part II).

Design, setting, and participants: Development of a ML model from a registry to predict NHR relying exclusively on prehospital predictors. Several models and imputation techniques were tested. Assess the feasibility to collect the predictors of the model in a customized smartphone app during prealert and generate a prediction in four level-1 trauma centers to compare the predictions to the gestalt of the trauma leader.

Main outcomes and measures: Part 1: Model output was NHR defined by 1) at least one RBC transfusion in resuscitation, 2) transfusion ≥ 4 RBC within 6 h, 3) any hemorrhage control procedure within 6 h or 4) death from hemorrhage within 24 h. The performance metric was the F4-score and compared to reference scores (RED FLAG, ABC). In part 2, the model and clinician prediction were compared with Likelihood Ratios (LR).

Results: From 36,325 eligible patients in the registry (Nov 2010-May 2022), 28,614 were included in the model development (Part 1). Median age was 36 [25-52], median ISS 13 [5-22], 3249/28614 (11%) corresponded to the definition of NHR. A XGBoost model with nine prehospital variables generated the best predictive performance for NHR according to the F4-score with a score of 0.76 [0.73-0.78]. Over a 3-month period (Aug-Oct 2022), 139 of 391 eligible patients were included in part II (38.5%), 22/139 with NHR. Clinician satisfaction was high, no workflow disruption observed and LRs comparable between the model and the clinicians.

Conclusions and relevance: The ShockMatrix pilot study developed a simple ML-enhanced NHR prediction tool demonstrating a comparable performance to clinical reference scores and clinicians. Collecting the predictor variables in real-time on prealert was feasible and caused no workflow disruption.

Keywords: Decision Support; Machine Learning; Prediction tool; Shock; Trauma.

Fig. 1

Study flowchart

Fig. 2

SHAP Diagram (XGBoost model). The SHAP value is used to report the weight of the variables in the model. Variables are ranked from the most important at the top (minimal SBP) to the less important at the bottom (sex). Variable influence on hemorrhagic shock prediction is represented from the right (positive) to left (negative) and the value of the observation is colored from red (highest for continuous variables and "yes” pour binomial variables) to blue (lowest for continuous variables and "No” pour binomial variables). As an example, “The minimal SPB is the better predictor of hemorrhagic shock (on the top) and the lower minimal SBP value (in blue) have the most positive impact on hemorrhagic shock prediction (right)”

Fig. 3

Confusion matrices

Fig. 4

Recall / Precision curve over threshold of the XGBoost model on the validation set. Adjusting the threshold for the positive cases from 11 to 5% improves the Sensitivity (Recall) to 0.90 while keeping the Precision over 0.25

Fig. 5

App assessment by participating clinicians in six dimensions, acceptability, ergonomics, data quality, protocol implementation, sample size, recruitment