Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Nov 1;158(11):1176-1183.
doi: 10.1001/jamasurg.2023.3673.

Risk-Adjusted Cumulative Sum for Early Detection of Hospitals With Excess Perioperative Mortality

Vivi W Chen  1   2 , Alexis P Chidi  3 , Yongquan Dong  1 , Peter A Richardson  1   4 , David A Axelrod  5 , Laura A Petersen  1   4 , Nader N Massarweh  6   7   8
Affiliations

Risk-Adjusted Cumulative Sum for Early Detection of Hospitals With Excess Perioperative Mortality

Vivi W Chen et al. JAMA Surg. .

Abstract

Importance: National surgical quality improvement programs lack tools for early detection of quality or safety concerns, which risks patient safety because of delayed recognition of poor performance.

Objective: To compare the risk-adjusted cumulative sum (CUSUM) with episodic evaluation for early detection of hospitals with excess perioperative mortality.

Design, setting, and participants: National, observational, hospital-level, comparative effectiveness study of 697 566 patients. Identification of hospitals with excess, risk-adjusted, quarterly 30-day mortality using observed to expected ratios (ie, current criterion standard in the Veterans Affairs Surgical Quality Improvement Program) was compared with the risk-adjusted CUSUM. Patients included in the study underwent a noncardiac operation at a Veterans Affairs hospital, had a record in the Veterans Affairs Surgical Quality Improvement Program (January 1, 2011, through December 31, 2016), and were aged 18 years or older.

Main outcome and measure: Number of hospitals identified as having excess risk-adjusted 30-day mortality.

Results: The cohort included 697 566 patients treated at 104 hospitals across 24 quarters. The mean (SD) age was 60.9 (13.2) years, 91.4% were male, and 8.6% were female. For each hospital, the median number of quarters detected with observed to expected ratios, at least 1 CUSUM signal, and more than 1 CUSUM signal was 2 quarters (IQR, 1-4 quarters), 8 quarters (IQR, 4-11 quarters), and 3 quarters (IQR, 1-4 quarters), respectively. During 2496 total quarters of data, outlier hospitals were identified 33.3% of the time (830 quarters) with at least 1 CUSUM signal within a quarter, 12.5% (311 quarters) with more than 1 CUSUM signal, and 11.0% (274 quarters) with observed to expected ratios at the end of the quarter. The CUSUM detection occurred a median of 49 days (IQR, 25-63 days) before observed to expected ratio reporting (1 signal, 35 days [IQR, 17-54 days]; 2 signals, 49 days [IQR, 26-61 days]; 3 signals, 58 days [IQR, 44-69 days]; ≥4 signals, 49 days [IQR, 42-69 days]; trend test, P < .001). Of 274 hospital quarters detected with observed to expected ratios, 72.6% (199) were concurrently detected by at least 1 CUSUM signal vs 42.7% (117) by more than 1 CUSUM signal. There was a dose-response relationship between the number of CUSUM signals in a quarter and the median observed to expected ratio (0 signals, 0.63; 1 signal, 1.28; 2 signals, 1.58; 3 signals, 2.08; ≥4 signals, 2.49; trend test, P < .001).

Conclusions: This study found that with CUSUM, hospitals with excess perioperative mortality can be identified well in advance of standard end-of-quarter reporting, which suggests episodic evaluation strategies fail to detect out-of-control processes and place patients at risk. Continuous performance evaluation tools should be adopted in national quality improvement programs to prevent avoidable patient harm.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Chen reported receiving grants from the Agency for Healthcare Research and Quality during the conduct of the study. Dr Chidi reported receiving personal fees from Intuitive Surgical outside the submitted work. Dr Axelrod reported receiving personal fees from CareDx and personal fees from Talaris outside the submitted work. Dr Petersen reported receiving grants from the National Institutes of Health during the conduct of the study and the Veterans Administration outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Example Cumulative Sum Run Charts With Corresponding Quarterly Observed to Expected (O-E) Ratios
Additional explanation of the figure is provided in eAppendix 1 in Supplement 1.
Figure 2.
Figure 2.. Distribution of Lag Time Before Observed to Expected Ratio (O-E) Identification, Stratified by Number of Cumulative Sum (CUSUM) Signals
See this image and copyright information in PMC

Comment in

References

    1. HR 505—Veterans’ Administration health-care amendments of 1985. Accessed June 6, 2023. https://www.congress.gov/bill/99th-congress/house-bill/505/text
    1. Khuri SF, Daley J, Henderson W, et al. ; National VA Surgical Quality Improvement Program . The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. Ann Surg. 1998;228(4):491-507. doi:10.1097/00000658-199810000-00006 - DOI - PMC - PubMed
    1. Birkmeyer JD, Shahian DM, Dimick JB, et al. . Blueprint for a new American College of Surgeons: National Surgical Quality Improvement Program. J Am Coll Surg. 2008;207(5):777-782. doi:10.1016/j.jamcollsurg.2008.07.018 - DOI - PubMed
    1. Khuri SF, Daley J, Henderson WG. The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg. 2002;137(1):20-27. doi:10.1001/archsurg.137.1.20 - DOI - PubMed
    1. Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program? an evaluation of all participating hospitals. Ann Surg. 2009;250(3):363-376. doi:10.1097/SLA.0b013e3181b4148f - DOI - PubMed

Publication types