Recalibration and External Validation of the Risk Analysis Index: A Surgical Frailty Assessment Tool

Abstract

Objective and background: The Risk Analysis Index (RAI) predicts 30-, 180-, and 365-day mortality based on variables constitutive of frailty. Initially validated, in a single-center Veteran hospital, we sought to improve model performance by recalibrating the RAI in a large, veteran surgical registry, and to externally validate it in both a national surgical registry and a cohort of surgical patients for whom RAI was measured prospectively before surgery.

Methods: The RAI was recalibrated among development and confirmation samples within the Veterans Affairs Surgical Quality Improvement Program (VASQIP; 2010-2014; N = 480,731) including major, elective noncardiac surgery patients to create the revised RAI (RAI-rev), comparing discrimination and calibration. The model was tested externally in the American College of Surgeons National Surgical Quality Improvement Program dataset (NSQIP; 2005-2014; N = 1,391,785), and in a prospectively collected cohort from the Nebraska Western Iowa Health Care System VA (NWIHCS; N = 6,856).

Results: Recalibrating the RAI significantly improved discrimination for 30-day [c = 0.84-0.86], 180-day [c = 0.81-0.84], and 365-day mortality [c = 0.78-0.82] (P < 0.001 for all) in VASQIP. The RAI-rev also had markedly better calibration (median absolute difference between observed and predicted 180-day mortality: decreased from 8.45% to 1.23%). RAI-rev was highly predictive of 30-day mortality (c = 0.87) in external validation with excellent calibration (median absolute difference between observed and predicted 30-day mortality: 0.6%). The discrimination was highly robust in men (c = 0.85) and women (c = 0.89). Discrimination also improved in the prospectively measured cohort from NWIHCS for 180-day mortality [c = 0.77 to 0.80] (P < 0.001).

Conclusions: The RAI-rev has improved discrimination and calibration as a frailty-screening tool in surgical patients. It has robust external validity in men and women across a wide range of surgical settings and available for immediate implementation for risk assessment and counseling in preoperative patients.

Figure 1:. Model Calibration: Observed vs. Predicted Mortality Across the Range of RAI Scores.

Predicted mortality for patients undergoing elective surgery was calculated using logistic regression with RAI scores as the sole independent variable. The predicted mortality for each RAI score is plotted against the observed mortality with 95% confidence intervals. The revised RAI showed significant improvement in model calibration, as demonstrated by improved c-statistic and overlap of the predicted mortality with observed mortality. Figure 1a: Observed vs. Predicted 180-day Mortality for RAI-A Original Score in Veterans Affairs Surgical Quality Improvement Program (VASQIP; c=0.813) Figure 1b: Observed vs. Predicted 180-day Mortality for RAI-rev Recalibrated Score in VASQIP (c=0.842) Figure 1c: Observed vs. Predicted 30-day Mortality for RAI-rev Recalibrated Score in American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP; c=0.87) Figure 1d: Observed vs. Predicted 30-day Mortality for RAI-C-rev Recalibrated Score in prospectively collected data at Nebraska Western Iowa Health Care System (NWIHCS; c=0.8)

Figure 2:. Model Calibration: Observed vs. Predicted 30 day Mortality Across the Range of RAI-rev Scores in (a) women [c=0.89] and (b) men [c=0.85].

Predicted mortality for patients undergoing elective surgery in the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) database was calculated using logistic regression with RAI-rev scores as the sole independent variable. The predicted mortality for each revised RAI score is plotted against the observed mortality with 95% confidence intervals.