Prediction cardiovascular deterioration in a paediatric intensive care unit (PicEWS): a machine learning modelling study of routinely collected health-care data

Abstract

Background: Paediatric intensive care medicine uses fine granular clinical data that describe substantial patient instability to make high-consequence decisions. However, these decisions are also hindered by clinical experts' ability to interpret longitudinal data along with recent and gradual changes in the vital sign data. Machine learning aided decisions can improve the identification of patient deterioration. Important prior work has predicted outcomes in paediatric intensive care units (PICUs), but has often used non-time series data without age normalisation. Most current work also aims to predict mortality, not potentially treatable clinical inflection points such as cardiovascular deterioration.

Methods: We extracted telemetry data, alongside laboratory and demographic data, from the Electronic Health Record (EHR) of patients admitted to the general PICU at Great Ormond Street Hospital, London (UK), between 1st April 2019 and 31st April 2021. We engineered deterioration monitoring variables into a smaller feature set using a generalisable pipeline. We calculated trend and variability, and used validated age-normalisation for input variables where appropriate. We compared neural network models, gradient-boosted decision trees (XGBoost), and a logistic regression model to predict cardiovascular deterioration within 12 h (defined as a rise in the paediatric Sequential Organ Failure Assessment (pSOFA) cardiovascular sub-score, rising plasma lactate if lactate ≥2 mmol/l, new extra-corporeal membrane oxygenation (ECMO) requirement, or death). We trained the models on a 70-15-15 percent train-test-validation split. We compared model compositions, including without trend, variability, and frequency of input to smaller models. We investigated feature importance using internal feature importance and Shapley Additive Explanation values. We compared the resulting paediatric intensive care early warning score (PicEWS) with the paediatric Sequential Organ Failure Assessment (pSOFA) score as the gold-standard.

Findings: 1167 patients were included out of a possible 1195. The best performing predictive model for PicEWS was XGBoost. PicEWS was able to predict cardiovascular deterioration 90% of the time, with fewer than two false alarms for every true alarm. For this model, the area under the precision-recall curve (AUPRC) was 0.552, and area under the receiver operator curve (AUROC) was 0.949. This outperformed pSOFA, which yielded over 10 false alarms per true alarm, with an AUPRC of 0.150 and AUROC of 0.715. The most important features for PicEWS included blood pressure, physiological markers such as bilirubin, and COMFORT score (a sedation and behavioural score used in paediatric intensive care). Feature variability was key to model performance. We demonstrated predictions on an individual patient to show model utility. The study showed that machine learning models can outperform current clinical best practice approaches. We use our model to provide insights into future improvements in clinical practice.

Interpretation: PicEWS outperforms current clinical modelling approaches to predict cardiovascular deterioration. The proposed data processing pipeline and machine learning method offer a clinically applicable decision-support model using age-stratified normal ranges and feature variability over time for the early detection of clinical deterioration in critically ill children.

Funding: The NIHR Great Ormond Street Biomedical Research Centre at UCL and the Great Ormond Street Hospital Children's Charity peer-reviewed grant award.

Keywords: Artificial intelligence; Critical & intensive & emergency care; Digital health; Paediatric intensive care.

Fig. 1

Summary of processing pipeline to make data ready for modelling. Data were extracted and consolidated to reduce input variable number (top panel). Feature engineering was performed, including age normalisation, and summary features produced. Finally, several different models were compared (bottom panel). NIBP: Non-invasive blood pressure, ABP: Arterial blood pressure, EHR: electronic health record, ECG: electrocardiogram, SpO2: oxygenated haemoglobin percentage saturation, HR: heart rate.

Fig. 2

(a) Performance metrics for the XGBoost models using different prediction windows. (b) Comparison of the PRC for the XGBoost model (top) and pSOFA with logistic regression model (bottom). These were produced by performing 10-fold cross-validation on the full dataset (including repeating the hyperparameter search each time) and the out of sample PRCs were then plotted. Across the full range of recall values the precision is substantially greater for the XGBoost model compared to pSOFA (with logistic regression link). AUPRC: Area under the precision-recall curve, AUROC: Area under the receiver-operator curve, F1: a weighted metric including precision and recall, PPV: Positive predictive value, PRC: Precision-recall curve, XGBoost: a type of gradient boosted decision tree, pSOFA: paediatric Sequential Organ Failure Assessment Score.

Fig. 3

SHAP values for mean arterial pressure (MAP), mean COMFORT score (sedation level), platelet count, serum sodium, serum chloride, serum strong ion gap (SIG, a measure of metabolic acidosis), serum bilirubin and serum creatinine (a–h). SHAP values plotted against input feature values for the top 20 values (i) and XGBoost internal feature importance for the top 20 features (j). SHAP: SHapley Additive exPlanation values, COMFORT: a behaviour and sedation score used in paediatric intensive care, XGBoost: a type of gradient boosted decision tree.

Fig. 4

Predicted probability of deterioration. (a) PicEWS model output plotted alongside mean arterial pressure (MAP), COMFORT Score and S:F Ratio, plotted against hour of admission. The red vertical line indicates the highest probability of deterioration and the grey box indicates the prediction interval. (b) SHAP values alongside the raw input values for this prediction. PicEWS: Paediatric Intensive Care Early Warning Score, MAP: Mean arterial Pressure, COMFORT score: a behaviour and sedation score used in paediatric intensive care, S:F Ratio: Oxygenated Haemoglobin Saturation (SpO2) to fraction inhaled oxygen (FiO2) ratio, SHAP: SHapley Additive exPlanation values.