Complete Blood Count and Monocyte Distribution Width-Based Machine Learning Algorithms for Sepsis Detection: Multicentric Development and External Validation Study

Abstract

Background: Sepsis is an organ dysfunction caused by a dysregulated host response to infection. Early detection is fundamental to improving the patient outcome. Laboratory medicine can play a crucial role by providing biomarkers whose alteration can be detected before the onset of clinical signs and symptoms. In particular, the relevance of monocyte distribution width (MDW) as a sepsis biomarker has emerged in the previous decade. However, despite encouraging results, MDW has poor sensitivity and positive predictive value when compared to other biomarkers.

Objective: This study aims to investigate the use of machine learning (ML) to overcome the limitations mentioned earlier by combining different parameters and therefore improving sepsis detection. However, making ML models function in clinical practice may be problematic, as their performance may suffer when deployed in contexts other than the research environment. In fact, even widely used commercially available models have been demonstrated to generalize poorly in out-of-distribution scenarios.

Methods: In this multicentric study, we developed ML models whose intended use is the early detection of sepsis on the basis of MDW and complete blood count parameters. In total, data from 6 patient cohorts (encompassing 5344 patients) collected at 5 different Italian hospitals were used to train and externally validate ML models. The models were trained on a patient cohort encompassing patients enrolled at the emergency department, and it was externally validated on 5 different cohorts encompassing patients enrolled at both the emergency department and the intensive care unit. The cohorts were selected to exhibit a variety of data distribution shifts compared to the training set, including label, covariate, and missing data shifts, enabling a conservative validation of the developed models. To improve generalizability and robustness to different types of distribution shifts, the developed ML models combine traditional methodologies with advanced techniques inspired by controllable artificial intelligence (AI), namely cautious classification, which gives the ML models the ability to abstain from making predictions, and explainable AI, which provides health operators with useful information about the models' functioning.

Results: The developed models achieved good performance on the internal validation (area under the receiver operating characteristic curve between 0.91 and 0.98), as well as consistent generalization performance across the external validation datasets (area under the receiver operating characteristic curve between 0.75 and 0.95), outperforming baseline biomarkers and state-of-the-art ML models for sepsis detection. Controllable AI techniques were further able to improve performance and were used to derive an interpretable set of diagnostic rules.

Conclusions: Our findings demonstrate how controllable AI approaches based on complete blood count and MDW may be used for the early detection of sepsis while also demonstrating how the proposed methodology can be used to develop ML models that are more resistant to different types of data distribution shifts.

Keywords: artificial intelligence; biomarker; clinical signs; clinical symptoms; complete blood count; controllable AI; data distribution; detection; development study; diagnostic; early detection; external validation; machine learning; machine learning model; medical machine learning; organ; organ dysfunction; sepsis; sepsis detection; validation study.

Figure 1

Performance (in terms of sensitivity, specificity, positive predictive value [PPV], and negative predictive value [NPV]) of the developed models, each represented by a colored circle, on the 6 considered datasets (represented on the y-axis), together with the corresponding 95% CIs. AR-ED: Arezzo, Emergency Department; DT: decision tree; LR: logistic regression; OGSA-ICU: Ospedale Galeazzi Sant’Ambrogio, Intensive Care Unit; PA-ED: Palermo, Emergency Department; PA-ICU: Palermo, Intensive Care Unit; PDR: Partial Decision Rule; PD-ICU: Padova, Intensive Care Unit; RF: random forest; SVM: support vector machine; UD-ED: Udine, Emergency Department; XGB: extreme gradient boosting.

Figure 2

Receiver operating characteristic curves for the models, along with their area under the receiver operating characteristic curve (AUC) values, on the Palermo, Emergency Department (PA-ED) internal validation dataset. In the plot we also report on the sensitivity and specificity of 4 baselines: binary thresholds based on the monocyte distribution width (MDW) [35] and C-reactive protein (CRP) [42], a binary threshold based on the Sepsis-Index parameter [43], the Conformal Multidimensional Prediction of Sepsis Risk model [25], and the support vector machine (SVM) model developed in the study by Aguirre and Urrechaga [31]. DT: decision tree; LR: logistic regression; PDR: Partial Decision Rule; RF: random forest; XGB: extreme gradient boosting.

Figure 3

Sensitivity–positive predictive value (PPV) curves for the models, along with their average PPV (A-PPV) values, on the Palermo, Emergency Department (PA-ED) internal validation dataset. The plot also shows the sensitivity and PPV of 4 baselines: binary thresholds based on the monocyte distribution width (MDW) [35] and C-reactive protein (CRP) [42], a binary threshold based on the Sepsis-Index parameter [43], the Conformal Multidimensional Prediction of Sepsis Risk model [25], and the support vector machine (SVM) model developed in the study by Aguirre and Urrechaga [31]. DT: decision tree; LR: logistic regression; PDR: Partial Decision Rule; RF: random forest; XGB: extreme gradient boosting.

Figure 4

Feature importances for the logistic regression (LR; left) and extreme gradient boosting (XGB; right) models. The feature importance for the LR model represents the coefficients of the induced linear model, shown as bar plots. For each feature, the color represents the sign of the coefficient. The width of the bar represents the magnitude of the coefficient: larger width denotes greater importance. The feature importance for the XGB model was computed through the Shapley Additive Explanations method and is represented in terms of violin plots: for each feature, red denotes high values, while blue denotes low values; values at the right of the middle vertical bar denote an increased confidence score for the positive class (sepsis), while values at the left denote a decreases confidence score. CRP: C-reactive protein; MCHC: mean corpuscular hemoglobin concentration; MCV: mean corpuscular volume; MDW: monocyte distribution width; NLR: neutrophils-lymphocytes ratio; RBC: red blood cell; WBC: white blood cell; HCT: hematocrit test; HGB: hemoglobin; PLT: platelet count.

Figure 5

Interpretable decision tree: the color of each node denotes the majority class for the corresponding subset of instances (blue: sepsis and orange: no sepsis). Each non–leaf node contains 3 data elements: a selected feature and corresponding threshold (top); the proportion of samples corresponding to that node (middle); and the proportion of samples belonging to each class (bottom). Leaf nodes contain 2 data elements: the proportion of samples corresponding to that node (top) and the proportion of samples belonging to each class (bottom). MDW: monocyte distribution width; RBC: red blood cell; WBC: white blood cell.

Figure 6

External performance diagram [40] for the extreme gradient boosting (XGB) model on the external datasets. The diagram illustrates the performance of the XGB model according to 3 different aspects: discrimination power (in terms of area under the receiver operating characteristic curve [AUC]), utility (in terms of standardized net benefit), and calibration (in terms of Brier score). The size of the ellipses associated with the datasets denotes the 95% CIs; the transparency of the ellipses denotes the achievement of the minimum sample size (the lower the transparency, the closer the sample size to the minimum sample size). The diagram has been produced with the tool available on the Metimeter website [58]. AR-ED: Arezzo, Emergency Department; OGSA-ICU: Ospedale Galeazzi Sant’Ambrogio, Intensive Care Unit; PA-ICU: Palermo, Intensive Care Unit; PD-ICU: Padova, Intensive Care Unit; UD-ED: Udine, Emergency Department.

Figure 7

Difference between the performance of cautious models and the corresponding standard models: values greater than 0 denote an improvement in performance in the cautious inference model as compared with the corresponding standard one. The 95% CIs were computed using the pooled SD. Each model is represented by a colored circle. AR-ED: Arezzo, Emergency Department, DT: decision tree; LR: logistic regression; OGSA-ICU: Ospedale Galeazzi Sant’Ambrogio, Intensive Care Unit, PA-ED: Palermo, Emergency Department; PA-ICU: Palermo, Intensive Care Unit; PDR: Partial Decision Rule; PD-ICU: Padova, Intensive Care Unit; RF: random forest; SVM: support vector machine; UD-ED: Udine, Emergency Department; XGB: extreme gradient boosting.