Combining physician's subjective and physiology-based objective mortality risk predictions

Abstract

Objective: None of the currently available physiology-based mortality risk prediction models incorporate subjective judgements of healthcare professionals, a source of additional information that could improve predictor performance and make such systems more acceptable to healthcare professionals. This study compared the performance of subjective mortality estimates by physicians and nurses with a physiology-based method, the Pediatric Risk of Mortality (PRISM) III. Then, healthcare provider estimates were combined with PRISM III estimates using Bayesian statistics. The performance of the Bayesian model was then compared with the original two predictions.

Design: Concurrent cohort study.

Setting: A tertiary pediatric intensive care unit at a university affiliated children's hospital.

Patients: Consecutive admissions to the pediatric intensive care unit.

Interventions: None.

Measurements and main results: For each of the 642 consecutive eligible patients, an exact mortality estimate and the degree of certainty (continuous scale from 1 to 5) associated with the estimate was collected from the attending, fellow, resident, and nurse responsible for the patient's care. Bayesian statistics were used to combine the PRISM III and certainty weighted subjective predictions to create a third Bayesian estimate of mortality. PRISM III discriminated survivors from nonsurvivors very well (area under curve [AUC], 0.924) as did the physicians and nurses (AUCs attendings, 0.953; fellows, 0.870; residents, 0.923; nurses, 0.935). Although the AUCs of the healthcare providers were not significantly different from the AUCs of PRISM III, the Bayesian AUCs were higher than both the healthcare providers' AUCs (p < or = .09 for all) and PRISM III AUCs. Similarly, the calibration statistics for the Bayesian estimates were superior to the calibration statistics for both the healthcare providers and PRISM III models.

Conclusions: The results of this study demonstrated that healthcare providers' subjective mortality predictions and PRISM III mortality predictions perform equally well. The Bayesian model that combined provider and PRISM III mortality predictions was more accurate than either provider or PRISM III alone and may be more acceptable to physicians. A methodology using subjective outcome predictions could be more relevant to individual patient decision support.