External validation and clinical utility of a prediction model for 6-month mortality in patients undergoing hemodialysis for end-stage kidney disease

Abstract

Background: End-stage kidney disease is associated with poor prognosis. Health care professionals must be prepared to address end-of-life issues and identify those at high risk for dying. A 6-month mortality prediction model for patients on dialysis derived in the United States is used but has not been externally validated.

Aim: We aimed to assess the external validity and clinical utility in an independent cohort in Canada.

Design: We examined the performance of the published 6-month mortality prediction model, using discrimination, calibration, and decision curve analyses.

Setting/participants: Data were derived from a cohort of 374 prevalent dialysis patients in two regions of British Columbia, Canada, which included serum albumin, age, peripheral vascular disease, dementia, and answers to the "the surprise question" ("Would I be surprised if this patient died within the next year?").

Results: The observed mortality in the validation cohort was 11.5% at 6 months. The prediction model had reasonable discrimination (c-stat = 0.70) but poor calibration (calibration-in-the-large = -0.53 (95% confidence interval: -0.88, -0.18); calibration slope = 0.57 (95% confidence interval: 0.31, 0.83)) in our data. Decision curve analysis showed the model only has added value in guiding clinical decision in a small range of threshold probabilities: 8%-20%.

Conclusion: Despite reasonable discrimination, the prediction model has poor calibration in this external study cohort; thus, it may have limited clinical utility in settings outside of where it was derived. Decision curve analysis clarifies limitations in clinical utility not apparent by receiver operating characteristic curve analysis. This study highlights the importance of external validation of prediction models prior to routine use in clinical practice.

Keywords: End-stage kidney disease; chronic kidney disease; dialysis; end-of-life care; palliative care; prediction model; prognosis.

Figure 1.

Cohort derivation.

Figure 2.

(a) ROC curve (left) and discrimination plot (right). The 6-month prediction model by Cohen et al. provided reasonable discrimination ability in the validation cohort. (b) Calibration plot, where the dark gray circles corresponded to the predicted risk for those who died (top) and alive (bottom) at 6 months. The 6-month prediction model by Cohen et al. was not well calibrated in the validation cohort. (c) Decision curve analysis. The 6-month prediction model by Cohen et al. only has added value in guiding clinical decision in a small range of threshold probabilities: 8%–20%, compared to both the “Treat-all” and “Treat-none” strategies.