Metrics Selection and Risk Adjustment Methods to Benchmark Inpatient Antibiotic Use

Abstract

Importance: The Centers for Disease Control and Prevention offers a standardized antimicrobial administration ratio (SAAR) as an evaluation metric for inpatient antibiotic use through rankings and peer comparisons (ie, benchmarking). However, the SAAR model only accounts for facility- and unit-level factors without considering the hierarchical nature of the health care data, and it does not directly reflect patient-level factors or stewardship efforts to avoid overly broad-spectrum therapy.

Objective: To examine the use of antimicrobial use risk adjustment methods and choice of basic metrics (eg, days of therapy [DOT] and days of antimicrobial spectrum coverage [DASC], which do not and do consider antimicrobial spectrum, respectively) in hospital benchmarking.

Design, setting, and participants: This retrospective cohort study was conducted using data from 117 acute care hospitals within the Veterans Health Administration (VHA) system. All patients admitted between January 1, 2021, and December 31, 2023, were included.

Main outcomes and measures: Monthly antibiotic use was measured with 2 basic metrics and risk adjustment models created using baseline data for 2021 to 2022. Hospitals were benchmarked for 2023 use via 3 methods: (1) unadjusted comparison, (2) risk adjustment with hospital- and unit-level factors with single-level negative binomial regression models (method 1, similar in approach to the SAAR), and (3) risk adjustment with hospital-, unit-, and patient-level factors with hierarchical zero-inflated negative binomial regression models (method 2).

Results: This study included 736 810 patients (median age, 70 [IQR, 61-76] years; 94.7% male). There was wide variability in unadjusted antibiotic use among hospitals (median, 477 [IQR, 420-523] DOT per 1000 days present [DP]; and median, 3115 [IQR, 2739-3602] DASC per 1000 DP). Risk adjustments with methods 1 and 2 resulted in moderate ranking changes, but there were only weak correlations between benchmarking results by the 2 methods (τB = 0.43 for DOT and 0.44 for DASC). The choice of basic metrics with or without consideration of antimicrobial spectrums (DOT vs DASC) had a modest correlation after risk adjustment (τB = 0.84).

Conclusions and relevance: In this cohort study of the nationwide VHA system, there were substantial differences in risk-adjusted benchmarking results between models with only hospital- and unit-level factors and models with hospital-, unit-, and patient-level factors. Future studies should evaluate whether these models with higher content validity also have better construct validity and can inform hospitals and stewardship programs about their objective performance compared with other programs.

Figure 1.. Changes in Hospital Benchmarking Results Based on Selection of Basic Metrics and Risk Adjustment Methods

In A and B, dashed lines indicate the median values of days of therapy (DOT) per 1000 days present (DP) or days of antimicrobial spectrum coverage (DASC) per 1000 DP. In C to F, dashed lines indicate an observed-to-expected (O:E) ratio or predicted-to-expected (P:E) ratio equal to 1. Each dot indicates 1 study hospital.

Figure 2.. Ranking Correlations by Selection of Basic Metrics and Risk Adjustment Methods

Kendall τ_B values are presented for correlations (strong, ≥0.7; moderate, 0.5 to <0.7; and weak, <0.5). DASC indicates days of antimicrobial spectrum coverage; DOT, days of therapy.

Figure 3.. Correlations for Hospital-Specific Random-Effect Estimates Between Count- and Zero-Inflation Components for DOT- and DASC-Based Models

Each dot indicates 1 study hospital. DASC indicates days of antimicrobial spectrum coverage; DOT, days of therapy; DP, days present; OR, odds ratio; RR, rate ratio.