External validation of a proprietary risk model for 1-year mortality in community-dwelling adults aged 65 years or older

Abstract

Objective: To examine the discrimination, calibration, and algorithmic fairness of the Epic End of Life Care Index (EOL-CI).

Materials and methods: We assessed the EOL-CI's performance by estimating area under the receiver operating characteristic curve (AUC), sensitivity, and positive and negative predictive values in community-dwelling adults ≥65 years of age in a single health system in the Southeastern United States. Algorithmic fairness was examined by comparing the model's performance across sex, race, and ethnicity subgroups. Using a machine learning approach, we also explored local re-calibration of the EOL-CI considering additional information on past hospitalizations and frailty.

Results: Among 215 731 patients (median age = 74 years, 57% female, 12% of Black race), 10% were classified as medium risk (15-44) and 3% as high risk (≥45) by the EOL-CI. The observed 1-year mortality rate was 3%. The EOL-CI had an AUC 0.82 for 1-year mortality, with a positive predictive value of 22%. Predictive performance was generally similar across sex and race subgroups, though the EOL-CI displayed better performance with increasing age and in older adults with 2 or more outpatient encounters in the past 24 months. Local re-calibration of the EOL-CI was required to provide absolute estimates of mortality risk, and calibration was further improved when the EOL-CI was augmented with data on inpatient hospitalizations and frailty.

Discussion: The EOL-CI demonstrates reasonable discrimination, albeit with better performance in older adults and in those with greater health system contact.

Conclusion: Local refinement and calibration of the EOL-CI score is required to provide direct estimates of prognosis, with the goal of making the EOL-CI a more a valuable tool at the point of care for identifying patients who would benefit from targeted palliative care interventions and proactive care planning.

Keywords: clinical decision-making; electronic health records; palliative care; prognosis.

Figure 1.

All-cause mortality stratified by the End-of-Life Care Index.

Figure 2.

Distribution of the Epic End-of-Life Care Index Score, with and without local modification, and resulting calibration with respect to observed 1-year mortality. (A) Calibration of Epic End-of-Life Index Score treating it as a probability (Initial), after re-calibration based on cross-validated Cox regression (Re-calibrated), after re-calibration based on cross-validated Cox regression including frailty and past inpatient hospitalizations as additional predictors (Re-calibrated + covariates), and finally utilizing a machine learning (ML) approach instead of Cox regression including frailty and past inpatient hospitalizations as additional predictors (Re-calibrated + covariates + ML). A total of 8 risk groups were used for each modeling approach as this was the largest number of groups that would allow unique cutpoints to be identified for the Epic End-of-Life Care Index. (B) The distribution of the Epic End-of-Life Index Score or predicted risk based on each of the re-calibrated models.

Figure 3.

Incidence of outpatient encounters for advance care planning by Epic End-of-Life Care Index Score. Analyses restricted to subgroup of older adults (N = 83 783) with a primary care provider within the Atrium Health-Wake Forest Baptist system.