Cross-validation performance of mortality prediction models

Abstract

Mortality prediction models hold substantial promise as tools for patient management, quality assessment, and, perhaps, health care resource allocation planning. Yet relatively little is known about the predictive validity of these models. We report here a comparison of the cross-validation performance of seven statistical models of patient mortality: (1) ordinary-least-squares (OLS) regression predicting 0/1 death status six months after admission; (2) logistic regression; (3) Cox regression; (4-6) three unit-weight models derived from the logistic regression, and (7) a recursive partitioning classification technique (CART). We calculated the following performance statistics for each model in both a learning and test sample of patients, all of whom were drawn from a nationally representative sample of 2558 Medicare patients with acute myocardial infarction: overall accuracy in predicting six-month mortality, sensitivity and specificity rates, positive and negative predictive values, and per cent improvement in accuracy rates and error rates over model-free predictions (i.e., predictions that make no use of available independent variables). We developed ROC curves based on logistic regression, the best unit-weight model, the single best predictor variable, and a series of CART models generated by varying the misclassification cost specifications. In our sample, the models reduced model-free error rates at the patient level by 8-22 per cent in the test sample. We found that the performance of the logistic regression models was marginally superior to that of other models. The areas under the ROC curves for the best models ranged from 0.61 to 0.63. Overall predictive accuracy for the best models may be adequate to support activities such as quality assessment that involve aggregating over large groups of patients, but the extent to which these models may be appropriately applied to patient-level resource allocation planning is less clear.