Development and Validation of a Machine Learning Model for Early Detection of Untreated Infection

Abstract

Background: Early diagnostic uncertainty for infection causes delays in antibiotic administration in infected patients and unnecessary antibiotic administration in noninfected patients.

Objective: To develop a machine learning model for the early detection of untreated infection (eDENTIFI), with the presence of infection determined by clinician chart review.

Derivation cohort: Three thousand three hundred fifty-seven adult patients hospitalized between 2006 and 2018 at two health systems in Illinois, United States.

Validation cohort: We validated in 1632 patients in a third Illinois health system using area under the receiver operating characteristic curve (AUC).

Prediction model: Using a longitudinal discrete-time format, we trained a gradient boosted machine model to predict untreated infection in the next 6 hours using routinely available patient demographics, vital signs, and laboratory results.

Results: eDENTIFI had an AUC of 0.80 (95% CI, 0.79-0.81) in the validation cohort and outperformed the systemic inflammatory response syndrome criteria with an AUC of 0.64 (95% CI, 0.64-0.65; p < 0.001). The most important features were body mass index, age, temperature, and heart rate. Using a threshold with a 47.6% sensitivity, eDENTIFI detected infection a median 2.0 hours (interquartile range, 0.9-5.2 hr) before antimicrobial administration, with a negative predictive value of 93.6%. Antibiotic administration guided by eDENTIFI could have decreased unnecessary IV antibiotic administration in noninfected patients by 10.8% absolute or 46.4% relative percentage points compared with clinicians.

Conclusion: eDENTIFI could both decrease the time to antimicrobial administration in infected patients and unnecessary antibiotic administration in noninfected patients. Further prospective validation is needed.

Figure 1.

Variable importance plot for early detection of untreated infection (eDENTIFI). This figure displays the ten most important predictor variables for eDENTIFI. Variable importance is calculated using the mean decrease in impurity and displayed on an adjusted scale of 0–100.

Figure 2.

Deciles of model predicted risk compared with the observed rates of infection and IV antibiotic administration. Patients were divided into ten equally sized groups according to their maximal predicted risk of untreated infection. Within each decile group, the observed rate of infection and IV antibiotic administration are compared. The rate of IV antibiotic administration was calculated from the ratio between the number of patients who received one or more doses of an IV antibiotic and the total number of patients within that decile group.