Veterans Affairs intensive care unit risk adjustment model: validation, updating, recalibration

Abstract

Background: A valid metric is critical to measure and report intensive care unit (ICU) outcomes and drive innovation in a national system.

Objectives: To update and validate the Veterans Affairs (VA) ICU severity measure (VA ICU).

Research design: A validated logistic regression model was applied to two VA hospital data sets: 36,240 consecutive ICU admissions to a stratified random sample of moderate and large hospitals in 1999-2000 (cohort 1) and 81,964 cases from 42 VA Medical Centers in fiscal years 2002-2004 (cohort 2). The model was updated by adding diagnostic groups and expanding the source of admission variables.

Measures: C statistic, Hosmer-Lemeshow goodness-of-fit statistic, and Brier's score measured predictive validity. Coefficients from the 1997 model were applied to predictors (fixed) in a logistic regression model. A 10 x 10 table compared cases with both VA ICU and National Surgical Quality Improvement Performance metrics. The standardized mortality ratios divided observed deaths by the sum of predicted mortality.

Results: The fixed model in both cohorts had predictive validity (cohort 1: C statistic = 0.874, Hosmer-Lemeshow goodness-of-fit C statistic chi-square = 72.5; cohort 2: 0.876, 307), as did the updated model (cohort 2: C statistic = 0.887, Hosmer-Lemeshow goodness-of-fit C statistic chi-square = 39). In 7,411 cases with predictions in both systems, the standardized mortality ratio was similar (1.04 for VA ICU, 1.15 for National Surgical Quality Improvement Performance), and 92% of cases matched (+/-1 decile) when ordered by deciles of mortality. The VA ICU standardized mortality ratio correlates with the National Surgical Quality Improvement Performance standardized mortality ratio (r2 = .74). Variation in discharge and laboratory practices may affect performance measurement.

Conclusion: The VA ICU severity model has face, construct, and predictive validity.