Aiming for Relevance

Abstract

Vital signs are crucial in intensive care units (ICUs). They are used to track the patient's state and to identify clinically significant changes. Predicting vital sign trajectories is valuable for early detection of adverse events. However, conventional machine learning metrics like RMSE often fail to capture the true clinical relevance of such predictions. We introduce novel vital sign prediction performance metrics that align with clinical contexts, focusing on deviations from clinical norms, overall trends, and trend deviations. These metrics are derived from empirical utility curves obtained in a previous study through interviews with ICU clinicians. We validate the metrics' usefulness using simulated and real clinical datasets (MIMIC and eICU). Furthermore, we employ these metrics as loss functions for neural networks, resulting in models that excel in predicting clinically significant events. This research paves the way for clinically relevant machine learning model evaluation and optimization, promising to improve ICU patient care.

Figure 1.

Normal range utility: heart rate example. The values between the red lines are the clinically normal range. The blue curve is the calculated normal range utility curve.

Figure 2.

Illustration of trend terms. Blue points represent the true vital sign measurements, and pink are predictions. The pink arrow represents the predicted trend ${\widehat{Y}}_t$, the light blue arrow represents the actual trend $Y_{t-n:t+m}$, and blue arrow is the expected trend $Y_{t-n:t}^{\text{'}}$.

Figure 3.

Simulation data: The top panel shows true values y_t (black) against XGBoost regressor step predictions (blue) for the simulated signal s₃. The middle and bottom panels display RMSE and U_td cost, respectively. The U_td cost is only sensitive to the sharp mismatch at the “drop” event.

Figure 4.

Simuation data: The top row presents RMSE over different types of events per model. On the bottom is the corresponding utility cost tested for the same models.

Figure 5.

LSTM model trained with RMSE loss for 6000 epochs (blue) vs. LSTM model trained with U_td component (orange) for 5000 epochs. With less training, the utility oriented model proves to be more accurate on sudden events.

Figure 6.

Simulation data: Performance of LSTMs trained with utility loss, in terms of RMSE over events of different types, as well as the corresponding utility metrics. Baseline LSTM is marked as ‘LSTM’, the Clinical range trained model is marked as ‘Range_LSTM’, the model trained for overall trend as ‘LSTM_Trend’ and the model trained for trend deviations is ‘LSTM_Dev’.

Figure 7.

eICU and MIMIC subsets blood pressure prediction task. The performance according to RMSE and each of the three utility costs is shown. The top row depicts these measures per model for the eICU dataset, and the bottom row depicts the same measures for the MIMIC dataset.

Figure 8.

Heartrate prediction over the eICU subset. Baseline model trained with RMSE loss (blue) vs. LSTM model trained additionally with U_td component (orange). The utility-optimized model captures more events. In the bottom row, the blue and orange bars overlap.

Figure 9.

Difference in RMSE over annotated events between vanilla LSTM model and Mixed model trained with utility costs. The task is heartrate prediction in the eICU dataset.