An empirical comparison of key statistical attributes among potential ICU quality indicators

Abstract

Objective: Good quality indicators should have face validity, relevance to patients, and be able to be measured reliably. Beyond these general requirements, good quality indicators should also have certain statistical properties, including sufficient variability to identify poor performers, relative insensitivity to severity adjustment, and the ability to capture what providers do rather than patients' characteristics. We assessed the performance of candidate indicators of ICU quality on these criteria. Indicators included ICU readmission, mortality, several length of stay outcomes, and the processes of venous-thromboembolism and stress ulcer prophylaxis provision.

Design: Retrospective cohort study.

Setting: One hundred thirty-eight U.S. ICUs from 2001-2008 in the Project IMPACT database.

Patients: Two hundred sixty-eight thousand eight hundred twenty-four patients discharged from U.S. ICUs.

Interventions: None.

Measurements and main results: We assessed indicators' (1) variability across ICU-years; (2) degree of influence by patient vs. ICU and hospital characteristics using the Omega statistic; (3) sensitivity to severity adjustment by comparing the area under the receiver operating characteristic curve (AUC) between models including vs. excluding patient variables, and (4) correlation between risk adjusted quality indicators using a Spearman correlation. Large ranges of among-ICU variability were noted for all quality indicators, particularly for prolonged length of stay (4.7-71.3%) and the proportion of patients discharged home (30.6-82.0%), and ICU and hospital characteristics outweighed patient characteristics for stress ulcer prophylaxis (ω, 0.43; 95% CI, 0.34-0.54), venous thromboembolism prophylaxis (ω, 0.57; 95% CI, 0.53-0.61), and ICU readmissions (ω, 0.69; 95% CI, 0.52-0.90). Mortality measures were the most sensitive to severity adjustment (area under the receiver operating characteristic curve % difference, 29.6%); process measures were the least sensitive (area under the receiver operating characteristic curve % differences: venous thromboembolism prophylaxis, 3.4%; stress ulcer prophylaxis, 2.1%). None of the 10 indicators was clearly and consistently correlated with a majority of the other nine indicators.

Conclusions: No indicator performed optimally across assessments. Future research should seek to define and operationalize quality in a way that is relevant to both patients and providers.

Figure 1

Variability of the eight dichotomous and two continuous quality indicators. VTE = venous thromboembolism.