A time-adjusted control chart for monitoring surgical outcome variations

Abstract

Background: Statistical Process Control (SPC) tools providing feedback to surgical teams can improve patient outcomes over time. However, the quality of routinely available hospital data used to build these tools does not permit full capture of the influence of patient case-mix. We aimed to demonstrate the value of considering time-related variables in addition to patient case-mix for detection of special cause variations when monitoring surgical outcomes with control charts.

Methods: A retrospective analysis from the French nationwide hospital database of 151,588 patients aged 18 and older admitted for colorectal surgery between January 1st, 2014, and December 31st, 2018. GEE multilevel logistic regression models were fitted from the training dataset to predict surgical outcomes (in-patient mortality, intensive care stay and reoperation within 30-day of procedure) and applied on the testing dataset to build control charts. Surgical outcomes were adjusted on patient case-mix only for the classical chart, and additionally on secular (yearly) and seasonal (quarterly) trends for the enhanced control chart. The detection of special cause variations was compared between those charts using the Cohen's Kappa agreement statistic, as well as sensitivity and positive predictive value with the enhanced chart as the reference.

Results: Within the 5-years monitoring period, 18.9% (28/148) of hospitals detected at least one special cause variation using the classical chart and 19.6% (29/148) using the enhanced chart. 59 special cause variations were detected overall, among which 19 (32.2%) discordances were observed between classical and enhanced charts. The observed Kappa agreement between those charts was 0.89 (95% Confidence Interval [95% CI], 0.78 to 1.00) for detecting mortality variations, 0.83 (95% CI, 0.70 to 0.96) for intensive care stay and 0.67 (95% CI, 0.46 to 0.87) for reoperation. Depending on surgical outcomes, the sensitivity of classical versus enhanced charts in detecting special causes variations ranged from 0.75 to 0.89 and the positive predictive value from 0.60 to 0.89.

Conclusion: Seasonal and secular trends can be controlled as potential confounders to improve signal detection in surgical outcomes monitoring over time.

Fig 1. Study flowchart.

Fig 2. Histograms of observed outcomes and the related curves of expected trends with classical or enhanced adjustments.

Expected rates of complications per quarter were calculated from the GEEs models initially fitted on the training dataset and then applied to the testing dataset. Models used for the construction of classical control charts were adjusted for case-mix variables only (age, gender, socioeconomic status, medical accessibility, emergency admission, hospital status, surgical procedure complexity, primary diagnosis, and comorbidities in dummies from the Elixhauser score). Models used for the construction of enhanced case-mix and time adjusted control charts were adjusted with the same set of variables, in addition to the year as a proxy for secular trends and the quarter as a proxy of seasonal variations.

Fig 3. Example of classical and enhanced control charts for two hospitals.

Crude observed rates (dotted black line) were monitored over 20 quarters for all three surgical outcomes. 2-SD warning limits (light green/red lines) and 3-SD control limits (bold green/red lines) were based on the central line (blue line) computed through the GEE models. A special cause variation related to a deterioration of surgical outcomes was detected in case of one single point beyond the 3-SD upper control limit (3-SD UCL), or 2 out of 3 consecutive points beyond the 2-SD upper warning limit (2-SD UWL). Conversely, a special cause variation related to an improvement of surgical outcomes was detected in case of one single point below the 3-SD lower control limit (3-SD LCL), or 2 out of 3 consecutive points below the 2-SD lower warning limit (2-SD LWL). The signal detection was considered at the first point beyond the limit when using the 2 out of 3 consecutive points rule. The two selected hospitals demonstrated discordances (encircled in pink) in interpretation of surgical outcome variations between classical and enhanced charts. Hospital A detected a special cause variation of increased mortality during the second quarter of 2018 using the time-adjusted chart (1 point above the 3-SD upper control limit, chart A2) but not the case-mix only adjusted chart (chart A1). Similarly, hospital B detected a special cause variation of decreased reoperation rate during the second quarter of 2015 using the time-adjusted chart (2 points out of 3 below the 2-SD lower warning limit, chart B6) but not the classical one (only 1 point out of 3 below the 2-SD warning limit, chart B5).