Performance of Risk Models for Antimicrobial Resistance in Adult Patients With Sepsis

Abstract

Importance: The results of prediction models that stratify patients with sepsis and risk of resistant gram-negative bacilli (GNB) infections inform treatment guidelines. However, these models do not extrapolate well across hospitals.

Objective: To assess whether patient case mix and local prevalence rates of resistance contributed to the variable performance of a general risk stratification GNB sepsis model for community-onset and hospital-onset sepsis across hospitals.

Design, setting, and participants: This was a retrospective cohort study conducted from January 2016 and October 2021. Adult patients with sepsis at 10 acute-care hospitals in rural and urban areas across Missouri and Illinois were included. Inclusion criteria were blood cultures indicating sepsis, having received 4 days of antibiotic treatment, and having organ dysfunction (vasopressor use, mechanical ventilation, increased creatinine or bilirubin levels, and thrombocytopenia). Analyses were completed in April 2024.

Exposure: The model included demographic characteristics, comorbidities, vital signs, laboratory values, procedures, and medications administered.

Main outcomes and measures: Culture results were stratified for ceftriaxone-susceptible GNB (SS), ceftriaxone-resistant but cefepime-susceptible GNB (RS), and ceftriaxone- and cefepime-resistant GNB (RR). Negative cultures and other pathogens were labeled SS. Deep learning models were developed separately for community-onset (patient presented with sepsis) and hospital-onset (sepsis developed ≥48 hours after admission) sepsis. The models were tested across hospitals and patient subgroups. Models were assessed using area under the receiver operating characteristic curve (AUROC) and area under precision recall curve (AUPRC).

Results: A total of 39 893 patients with 85 238 sepsis episodes (43 207 [50.7%] community onset; 42 031 [48.3%] hospital onset) were included. Median (IQR) age was 65 (54-74) years, 21 241 patients (53.2%) were male, and 18 830 (47.2%) had a previous episode of sepsis. RS contributed to 3.9% (1667 episodes) and 5.7% (2389 episodes) of community-onset and hospital-onset sepsis episodes, respectively, and RR contributed to 1.8% (796 episodes) and 3.9% (1626 episodes), respectively. Previous infections and exposure to antibiotics were associated with the risk of resistant GNB. For example, in community-onset sepsis, 375 RR episodes (47.1%), 420 RS episodes (25.2%) and 3483 of 40 744 (8.5%) SS episodes were among patients with resistance to antimicrobial drugs (P < .001). The AUROC and AUPRC results varied across hospitals and patient subgroups for both community-onset and hospital-onset sepsis. AUPRC values correlated with the prevalence rates of resistant GNB (R = 0.79; P = .001).

Conclusions and relevance: In this cohort study of 39 893 patients with sepsis, variable model performance was associated with prevalence rates of antimicrobial resistance rather than patient case mix. This variability suggests caution is needed when using generalized models for predicting resistant GNB etiologies in sepsis.

Figure 1.. Ten Most Important Features for Gradient-Boosting Model Performance in Community-Onset and Hospital-Onset Sepsis

Feature importance in descending order is shown on the vertical axis. Plotted points represent the distinct value for each infection analyzed with color coding if the value was high (red) or low (blue). Shapley additive explanation (SHAP) values displayed on the horizontal axis reflect whether a value was associated with increased or decreased risk. ALT indicates alanine transaminase; BMI, body mass index; E coli, Escherichia coli; ESBL, extended spectrum beta-lactamase; GNB, gram-negative bacilli; P aeruginosa, Pseudomonas aeruginosa; PEEP, positive end-expiratory pressure; RR, ceftriaxone-resistant and cefepime-resistant gram-negative bacilli; RS, ceftriaxone-resistant and cefepime-susceptible gram-negative bacilli; SS, ceftriaxone- and cefepime-susceptible gram-negative bacilli; and WBC, white blood cell.

Figure 2.. Deep Learning Model Performance Across Patient Subgroups of Interest and Hospitals

The number of sepsis episodes and the positive rates of gram-negative bacilli (GNB) isolates resistant to ceftriaxone and susceptible to cefepime as well as GNB isolates resistant to both ceftriaxone and cefepime are shown. C, Whiskers indicate ±1 SD. Subgroups are as follows: 0, entire cohort; 1, patients aged 65 years or older; 2, patients younger than 65 years; 3, patients with history of bacterial pneumonia; 4, patients with history of other sepsis; 5, patients with history of disease-causing GNB such as Escherichia coli, Klebsiella spp, and Pseudomonas aeruginosa; 6, patients with history of antibiotic-resistant microbes; 7, patients with no history of bacterial pneumonia, other sepsis, disease-causing GNB, or antibiotic resistant microbes; 8, patients with hematological malignant neoplasms; 9, patients who underwent transplant; 10, patients with alcoholic cirrhosis; 11, patients with septic shock; and 12, patients who underwent intubation. AUPRC indicates area under the precision recall curve.

Figure 3.. Correlation Plots Between Prevalence Rate of Gram-Negative Bacilli (GNB) Isolates Resistant to Ceftriaxone and Susceptible to Cefepime (RS) and GNB Isolates Resistant to Both Ceftriaxone and Cefepime (RR) and Sample Size With Area Under the Precision Recall Curve (AUPRC) Across Patient Subgroups

Analyses were repeated after excluding patient subgroups that contributed less than 0.2 fraction in the dataset without improved R coefficient or P value.