Factors affecting the performance of the models in the Mortality Probability Model II system and strategies of customization: a simulation study

Abstract

Objectives: To examine the impact of hospital mortality and intensive care unit (ICU) size on the performance of the Mortality Probability Model II system for use in quality assessment, and to examine the ability of model customization to produce accurate estimates of hospital mortality to characterize patients by severity of illness for clinical trials.

Design: Prospective evaluation of model performance, using retrospective data.

Setting: Data for the simulation were assembled from six adult medical and surgical ICUs in Massachusetts and New York.

Patients: Consecutive admissions (n = 4,224) to the Massachusetts and New York ICUs were studied. The mortality rate in the database was 18.7%.

Interventions: A computer simulation of several different hospital mortality rates and ICU sample sizes, using a multicenter database of consecutive ICU admissions, was utilized. We simulated 20 different mortality rates by randomly changing the outcomes at hospital discharge from "survived" to "deceased" and from "deceased" to "survived". Four sample size simulations used 75%, 50%, 25%, and 10% of the database. Ten replications of each mortality rate and samples size were constructed, and model calibration and discrimination were assessed for each replication. Model coefficients were customized, using logistic regression.

Measurements and main results: Vital status at hospital discharge was the outcome measure among the ICU patient population. Model performance was assessed using the Hosmer-Lemeshow C statistic for calibration, and the area under the receiver operating characteristic curve for discrimination. Goodness-of-fit tests and receiver operating characteristic curve areas demonstrated that the models were sensitive to differences in hospital mortality, indicating that they are useful quality assurance tools. Goodness-of-fit tests were more sensitive than the receiver operating characteristic curve areas. The further the hospital mortality rate diverged from the original rate, the worse the performance of the model. Sample size had an impact on these results. The smaller the sample size, the less likely the model was to perform poorly. Model coefficients were successfully customized to demonstrate that improved model performance can be achieved when necessary for clinical trial stratification.

Conclusion: Mortality Probability Model II models can be used to assess quality of care in ICUs, but the size of the sample should be considered when assessing calibration and discrimination.