Evaluating equity in performance of an electronic health record-based 6-month mortality risk model to trigger palliative care consultation: a retrospective model validation analysis

Abstract

Objective: Evaluate predictive performance of an electronic health record (EHR)-based, inpatient 6-month mortality risk model developed to trigger palliative care consultation among patient groups stratified by age, race, ethnicity, insurance and socioeconomic status (SES), which may vary due to social forces (eg, racism) that shape health, healthcare and health data.

Design: Retrospective evaluation of prediction model.

Setting: Three urban hospitals within a single health system.

Participants: All patients ≥18 years admitted between 1 January and 31 December 2017, excluding observation, obstetric, rehabilitation and hospice (n=58 464 encounters, 41 327 patients).

Main outcome measures: General performance metrics (c-statistic, integrated calibration index (ICI), Brier Score) and additional measures relevant to health equity (accuracy, false positive rate (FPR), false negative rate (FNR)).

Results: For black versus non-Hispanic white patients, the model's accuracy was higher (0.051, 95% CI 0.044 to 0.059), FPR lower (-0.060, 95% CI -0.067 to -0.052) and FNR higher (0.049, 95% CI 0.023 to 0.078). A similar pattern was observed among patients who were Hispanic, younger, with Medicaid/missing insurance, or living in low SES zip codes. No consistent differences emerged in c-statistic, ICI or Brier Score. Younger age had the second-largest effect size in the mortality prediction model, and there were large standardised group differences in age (eg, 0.32 for non-Hispanic white versus black patients), suggesting age may contribute to systematic differences in the predicted probabilities between groups.

Conclusions: An EHR-based mortality risk model was less likely to identify some marginalised patients as potentially benefiting from palliative care, with younger age pinpointed as a possible mechanism. Evaluating predictive performance is a critical preliminary step in addressing algorithmic inequities in healthcare, which must also include evaluating clinical impact, and governance and regulatory structures for oversight, monitoring and accountability.

Keywords: decision support, computerized; evaluation methodology; information technology.

Figure 1

Model predictive performance for each subgroup, TRIPOD-recommended metrics. Age quartiles comprised: youngest (18.1–47.8 years), second quartile (47.8–60.8 years), third quartile (60.8–71.2 years) and oldest (71.2 to ≥90 years). Zip code level median household income quartiles comprised: lowest quartile ($11 269 to $33 117), second quartile ($33 117–$58 784), third quartile ($58 784–$80 363) and highest quartile ($80 363–$225 598). Zip code level educational attainment (proportion of residents ≥25 years old who completed at least a bachelor’s degree, inclusive of all higher levels) quartiles comprised: lowest quartile (0%–21.9%), second quartile (21.9%–28.6%), third quartile (28.6%–48.7%) and highest quartile (48.8%–100%). ICI, integrated calibration index; TRIPOD, Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis.

Figure 2

Model predictive performance for each subgroup, health equity-relevant metrics. Age quartiles comprised: youngest (18.1–47.8 years), second quartile (47.8–60.8 years), third quartile (60.8–71.2 years) and oldest (71.2 to ≥90 years). Zip code level median household income quartiles comprised: lowest quartile ($11 269 to $33 117), second quartile ($33 117 to $58 784), third quartile ($58 784 to $80 363) and highest quartile ($80 363 to $225 598). Zip code level educational attainment (proportion of residents ≥25 years old who completed at least a bachelor’sdegree, inclusive of all higher levels) quartiles comprised: lowest quartile (0%–21.9%), second quartile (21.9%–28.6%), third quartile (28.6%–48.7%) and highest quartile (48.8%–100%). FPR, false positive rate; FNR, false negative rate.

Figure 3

Original mortality risk model predictor coefficients versus the standardised mean difference in predictors, non-Hispanic white versus black and Asian patients. All 34 predictors included in the original EHR-based mortality risk model are represented in this plot. Variable coefficient estimates are represented on the x-axis; standardised mean difference in predictors (difference between the two group means divided by the SD of the variable) is represented on the y-axis. The standardised mean differences were all calculated via reference group minus selected subgroup (eg, non-Hispanic white patient mean of a selected predictor – black patient mean of a selected predictor). The predictor contributes to predictive performance disparities if (1) The effect size is large and positive and the standardised mean difference is large and positive) or (2) The effect size is large and negative and the standardised mean difference is large and negative. EHR, electronic health record.