Challenges of Modeling Outcomes for Surgical Infections: A Word of Caution

Abstract

Background: We developed a novel analytic tool for colorectal deep organ/space surgical site infections (C-OSI) prediction utilizing both institutional and extra-institutional American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) data. Methods: Elective colorectal resections (2006-2014) were included. The primary end point was C-OSI rate. A Bayesian-Probit regression model with multiple imputation (BPMI) via Dirichlet process handled missing data. The baseline model for comparison was a multivariable logistic regression model (generalized linear model; GLM) with indicator parameters for missing data and stepwise variable selection. Out-of-sample performance was evaluated with receiver operating characteristic (ROC) analysis of 10-fold cross-validated samples. Results: Among 2,376 resections, C-OSI rate was 4.6% (n = 108). The BPMI model identified (n = 57; 56% sensitivity) of these patients, when set at a threshold leading to 80% specificity (approximately a 20% false alarm rate). The BPMI model produced an area under the curve (AUC) = 0.78 via 10-fold cross- validation demonstrating high predictive accuracy. In contrast, the traditional GLM approach produced an AUC = 0.71 and a corresponding sensitivity of 0.47 at 80% specificity, both of which were statstically significant differences. In addition, when the model was built utilizing extra-institutional data via inclusion of all (non-Mayo Clinic) patients in ACS-NSQIP, C-OSI prediction was less accurate with AUC = 0.74 and sensitivity of 0.47 (i.e., a 19% relative performance decrease) when applied to patients at our institution. Conclusions: Although the statistical methodology associated with the BPMI model provides advantages over conventional handling of missing data, the tool should be built with data specific to the individual institution to optimize performance.

Keywords: colorectal; modeling; organ space infection; risk prediction.

FIG. 1.

Distribution of predictive risk for C-OSI among the entire cohort. The dashed line indicates the set cut-off value of probaility = 0.060 leading to a 20% false alarm rate. C-OSI = colorectal deep organ/space infection; SSI = surgical site infection.

FIG. 2.

Confusion matrix based on 10-fold cross validation for the proposed approach (BPMI) for predicting deep/organ space surgical site infection (C-OSI) at a 10% false alarm threshold (i.e., setting specificity = 0.90). BPMI = Bayesian-Probit regression model with multiple-imputation; C-OSI = colorectal deep/organ space infection; TN = true-negative; FN = false-negative; FP = false-positive;, TP = true-positive; NPV = negative predictive value; PPV = positive predictive value.

FIG. 3.

The ROC based on 10-fold cross-validation for the proposed approach (BPMI) and the standard approach (GLM FAC) for predicting deep organ/space surgical site infection. The ROC curves of the BPMI model utilizing the Mayo Clinic-only data pool (solid line), the BPMI model utilizing the national NSQIP data pool to predict C-OSI on Mayo-Clinic only data (dashed line), and the GLM FAC model built on Mayo Clinic-only data (dotted line). ROC = receiver operator characteristic; BPMI = Bayesian-Probit regression model with multiple-imputation; GLM FAC = stepwsie logistic regression treating missingness with factor/indicator parameters; C-OSI = colorectal deep organ/space infection; AUC = area under curve.

FIG. 4.

Parameter importance summary for all predictors with a posterior inclusion probability above 0.3. Representation of clinical variables and their relative variable importance. Calculations are based on the posterior probability that the effect is non-zero. Posterior is the posterior probability that the parameter was included in the model. Estimate is the posterior mean estimate of the coeficient in the probit model. Ninety-five percent confidence intervals along with estimate are presented graphically on the far right. Only parameters with a posterior inclusion probability above 0.3 are displayed. ASA = American Society of Anesthesiologists; DM = diabetes mellitus; wound = wound class type 3 (contaminated) and 4 (infectious); PT = prothrombin; INR = international normalized ratio.

FIG. 5.

Proposed monitoring tool display. The risk of SSI along with uncertainty limits are graphically presented, along with the factors that most contributed to an elevated risk prediction. The 95% prediction band quantifies the uncertainty in the risk score because of missing factors. Missing factors that resulted in the largest proportion of this uncertainty are displayed on the right. If the uncertainty is large enough, some of the missing factors may be obtainable, e.g., certain laboratory tests. WBC = white blood cell count; SGOT = serum glutamic-oxaloacetic transaminase.