Effects of Neighborhood-level Data on Performance and Algorithmic Equity of a Model That Predicts 30-day Heart Failure Readmissions at an Urban Academic Medical Center

Abstract

Background: Socioeconomic data may improve predictions of clinical events. However, owing to structural racism, algorithms may not perform equitably across racial subgroups. Therefore, we sought to compare the predictive performance overall, and by racial subgroup, of commonly used predictor variables for heart failure readmission with and without the area deprivation index (ADI), a neighborhood-level socioeconomic measure.

Methods and results: We conducted a retrospective cohort study of 1316 Philadelphia residents discharged with a primary diagnosis of congestive heart failure from the University of Pennsylvania Health System between April 1, 2015, and March 31, 2017. We trained a regression model to predict the probability of a 30-day readmission using clinical and demographic variables. A second model also included the ADI as a predictor variable. We measured predictive performance with the Brier Score (BS) in a held-out test set. The baseline model had moderate performance overall (BS 0.13, 95% CI 0.13-0.14), and among White (BS 0.12, 95% CI 0.12-0.13) and non-White (BS 0.13, 95% CI 0.13-0.14) patients. Neither performance nor algorithmic equity were significantly changed with the addition of the ADI.

Conclusions: The inclusion of neighborhood-level data may not reliably improve performance or algorithmic equity.

Keywords: Algorithmic equity; congestive heart failure; hospital readmission.

Figure 1:

Overview of the study design and methods.

Figure 2:

Model performance in aggregate and by race for each model type. Abbreviations: BS = Brier score.

Figure 3:

Model performance with an increasing number of input variables. Each model was trained 50 times with each number of variables. A different set of variables was randomly drawn for each iteration within the same number of variables.

Figure 4:

Model performance with an increasing the number of observations. Each model was trained 50 times with each number of observations.

Figure 5:

The positive predictive value for the elastic net (top) and gradient boosting machine (bottom) models varies by race across different predictive thresholds. The absence of an estimate at a particular threshold indicates there were no predictions for that group above that threshold and thus the positive predictive value could not be calculated. Abbreviations: ADI = Area Deprivation Index.