Identifying unusual performance in Australian and New Zealand intensive care units from 2000 to 2010

Abstract

Background: The Australian and New Zealand Intensive Care Society (ANZICS) Adult Patient Database (APD) collects voluntary data on patient admissions to Australian and New Zealand intensive care units (ICUs). This paper presents an in-depth statistical analysis of risk-adjusted mortality of ICU admissions from 2000 to 2010 for the purpose of identifying ICUs with unusual performance.

Methods: A cohort of 523,462 patients from 144 ICUs was analysed. For each ICU, the natural logarithm of the standardised mortality ratio (log-SMR) was estimated from a risk-adjusted, three-level hierarchical model. This is the first time a three-level model has been fitted to such a large ICU database anywhere. The analysis was conducted in three stages which included the estimation of a null distribution to describe usual ICU performance. Log-SMRs with appropriate estimates of standard errors are presented in a funnel plot using 5% false discovery rate thresholds. False coverage-statement rate confidence intervals are also presented. The observed numbers of deaths for ICUs identified as unusual are compared to the predicted true worst numbers of deaths under the model for usual ICU performance.

Results: Seven ICUs were identified as performing unusually over the period 2000 to 2010, in particular, demonstrating high risk-adjusted mortality compared to the majority of ICUs. Four of the seven were ICUs in private hospitals. Our three-stage approach to the analysis detected outlying ICUs which were not identified in a conventional (single) risk-adjusted model for mortality using SMRs to compare ICUs. We also observed a significant linear decline in mortality over the decade. Distinct yearly and weekly respiratory seasonal effects were observed across regions of Australia and New Zealand for the first time.

Conclusions: The statistical approach proposed in this paper is intended to be used for the review of observed ICU and hospital mortality. Two important messages from our study are firstly, that comprehensive risk-adjustment is essential in modelling patient mortality for comparing performance, and secondly, that the appropriate statistical analysis is complicated.

Figure 1

Yearly and weekly respiratory seasonal effects across Australia and New Zealand in 2010. Estimated annual cycle from 1 January (day 1) to 31 December (day 365) and weekly cycle Saturday to Friday, for Australian States and Territories and New Zealand. The seasonal effects are for 2010 respiratory patients, conditional on the estimated Stage 1 model. Queensland (QLD) and the Australian Capital Territory (ACT) have significantly earlier (March) and later (August) peak mortalities compared to New South Wales (NSW), which peaks in June. QLD and Tasmania (TAS) have weekly peak mortalities associated with admissions on Mondays and Wednesdays respectively, compared to NSW which has weekly peak mortality associated with Saturday admissions. No other cycles differ significantly from NSW. Legend: NT, Northern Territory; SA, South Australia; VIC, Victoria; WA, Western Australia; NZ, New Zealand.

Figure 2

Kernel density plot of total ICU volume over 2000-2010. Large tick-marks indicate volumes of ICUs deemed to be potentially unusual at Stage 1 of the analysis.

Figure 3

Funnel plot of log-SMRs versus effective sample size for each ICU from Stage 2 of the analysis. The funnels correspond to 95% classical limits (dashed lines) not adjusted for multiple testing, the Bonferroni limits controlling the FWER at 5% (dotted lines), and limits controlling the FDR at 5% (solid lines). Potentially unusual ICUs are marked with their random identifying numbers. The effective sample size is the estimated variance of the log-SMR as a fraction of the total variance. Legend: SMR: standardised mortality ratio. FDR: false discovery rate. FWER: family-wise error rate.

Figure 4

Five percent false coverage-statement rate confidence interval estimates for all 144 ICUs. The seven 5% FDR-selected ICUs identified as unusual are highlighted in bold on the left; these ICUs have confidence intervals with the correct FCR rate of 0.05. The remainder have FCR coverage of at most 0.05 for all parameters because the FCR offers marginal coverage of at least 0.95. The ICUs are ordered from those with the smallest to the largest p-values. Legend: FDR: false discovery rate. FCR: false coverage-statement rate.

Figure 5

Predicted distributions of the ‘true worst’ numbers of deaths for each of the seven unusual ICUs. Each subplot shows the simulated probability density functions for the predicted number of deaths and predicted ‘true worst’ number of deaths for the seven ICUs identified as unusual. Each simulation is based on 50,000 replications. The observed number of deaths is indicated by the solid vertical line in each case. The plots are presented in order of the true worst (ICU 140), second true worst (ICU 16), and so on, up to the seventh true worst (ICU 54).

Figure 6

Yearly estimated log-SMRs plotted over time for the seven unusual ICUs. In each case, the yearly log-SMRs are shown by bullets joined by solid lines with ± (plus and minus) two standard errors marked by open circles joined by dashed lines. The ICUs are presented in random-identity number order.