Missing data in the American College of Surgeons National Surgical Quality Improvement Program are not missing at random: implications and potential impact on quality assessments

Abstract

Background: Studying risk-adjusted outcomes in health care relies on statistical approaches to handling missing data. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides risk-adjusted assessments of surgical programs, traditionally imputing certain missing data points using a single round of multivariable imputation. Such imputation assumes that data are missing at random-without systematic bias-and does not incorporate estimation uncertainty. Alternative approaches, including using multiple imputation to incorporate uncertainty or using an indicator of missingness, can enhance robustness of evaluations.

Study design: One year of de-identified data from the ACS NSQIP, representing 117 institutions and 106,113 patients, was analyzed. Using albumin variables as the missing data modeled, several imputation/adjustment models were compared, including the traditional NSQIP imputation, a new single imputation, a multiple imputation, and use of a missing indicator.

Results: Coefficients for albumin values changed under new single imputation and multiple imputation approaches. Multiple imputation resulted in increased standard errors, as expected. An indicator of missingness was highly explanatory, disproving the missing-at-random assumption. The effects of changes in approach differed for different outcomes, such as mortality and morbidity, and effects were greatest in smaller datasets. However, ultimate changes in patient risk assessment and institutional assessment were minimal.

Conclusions: Newer statistical approaches to modeling missing (albumin) values result in noticeable statistical distinctions, including improved incorporation of imputation uncertainty. In addition, the missing-at-random assumption is incorrect for albumin. Despite these findings, effects on institutional assessments are small. Although effects can be most important with smaller data-sets, the current approach to imputing missing values in the ACS NSQIP appears reasonably robust.