Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models

Abstract

Objective: To compare risk prediction models for death in hospital based on an administrative database with published results based on data derived from three national clinical databases: the national cardiac surgical database, the national vascular database and the colorectal cancer study.

Design: Analysis of inpatient hospital episode statistics. Predictive model developed using multiple logistic regression.

Setting: NHS hospital trusts in England.

Patients: All patients admitted to an NHS hospital within England for isolated coronary artery bypass graft (CABG), repair of abdominal aortic aneurysm, and colorectal excision for cancer from 1996-7 to 2003-4.

Main outcome measures: Deaths in hospital. Performance of models assessed with receiver operating characteristic (ROC) curve scores measuring discrimination (<0.7=poor, 0.7-0.8=reasonable, >0.8=good) and both Hosmer-Lemeshow statistics and standardised residuals measuring goodness of fit.

Results: During the study period 152 523 cases of isolated CABG with 3247 deaths in hospital (2.1%), 12 781 repairs of ruptured abdominal aortic aneurysm (5987 deaths, 46.8%), 31 705 repairs of unruptured abdominal aortic aneurysm (3246 deaths, 10.2%), and 144,370 colorectal resections for cancer (10,424 deaths, 7.2%) were recorded. The power of the complex predictive model was comparable with that of models based on clinical datasets with ROC curve scores of 0.77 (v 0.78 from clinical database) for isolated CABG, 0.66 (v 0.65) and 0.74 (v 0.70) for repairs of ruptured and unruptured abdominal aortic aneurysm, respectively, and 0.80 (v 0.78) for colorectal excision for cancer. Calibration plots generally showed good agreement between observed and predicted mortality.

Conclusions: Routinely collected administrative data can be used to predict risk with similar discrimination to clinical databases. The creative use of such data to adjust for case mix would be useful for monitoring healthcare performance and could usefully complement clinical databases. Further work on other procedures and diagnoses could result in a suite of models for performance adjusted for case mix for a range of specialties and procedures.

Fig 1 ROC curve areas comparing simple (year, age, sex), intermediate (including method of admission), and complex models derived from HES with best model derived from clinical databases for index procedures (CABG=coronary artery bypass graft; AAA=abdominal aortic aneurysm)

Fig 2 Calibration plots for complex risk prediction models based on HES data for index procedures showing observed number of deaths against predicted based on validation set (CABG=coronary artery bypass graft; AAA=abdominal aortic aneurysm)