Discriminative Accuracy of Physician and Nurse Predictions for Survival and Functional Outcomes 6 Months After an ICU Admission

Abstract

Importance: Predictions of long-term survival and functional outcomes influence decision making for critically ill patients, yet little is known regarding their accuracy.

Objective: To determine the discriminative accuracy of intensive care unit (ICU) physicians and nurses in predicting 6-month patient mortality and morbidity, including ambulation, toileting, and cognition.

Design, setting, and participants: Prospective cohort study conducted in 5 ICUs in 3 hospitals in Philadelphia, Pennsylvania, and enrolling patients who spent at least 3 days in the ICU from October 2013 until May 2014 and required mechanical ventilation, vasopressors, or both. These patients' attending physicians and bedside nurses were also enrolled. Follow-up was completed in December 2014.

Main outcomes and measures: ICU physicians' and nurses' binary predictions of in-hospital mortality and 6-month outcomes, including mortality, return to original residence, ability to toilet independently, ability to ambulate up 10 stairs independently, and ability to remember most things, think clearly, and solve day-to-day problems (ie, normal cognition). For each outcome, physicians and nurses provided a dichotomous prediction and rated their confidence in that prediction on a 5-point Likert scale. Outcomes were assessed via interviews with surviving patients or their surrogates at 6 months. Discriminative accuracy was measured using positive and negative likelihood ratios (LRs), C statistics, and other operating characteristics.

Results: Among 340 patients approached, 303 (89%) consented (median age, 62 years [interquartile range, 53-71]; 57% men; 32% African American); 6-month follow-up was completed for 299 (99%), of whom 169 (57%) were alive. Predictions were made by 47 physicians and 128 nurses. Physicians most accurately predicted 6-month mortality (positive LR, 5.91 [95% CI, 3.74-9.32]; negative LR, 0.41 [95% CI, 0.33-0.52]; C statistic, 0.76 [95% CI, 0.72-0.81]) and least accurately predicted cognition (positive LR, 2.36 [95% CI, 1.36-4.12]; negative LR, 0.75 [95% CI, 0.61-0.92]; C statistic, 0.61 [95% CI, 0.54-0.68]). Nurses most accurately predicted in-hospital mortality (positive LR, 4.71 [95% CI, 2.94-7.56]; negative LR, 0.61 [95% CI, 0.49-0.75]; C statistic, 0.68 [95% CI, 0.62-0.74]) and least accurately predicted cognition (positive LR, 1.50 [95% CI, 0.86-2.60]; negative LR, 0.88 [95% CI, 0.73-1.06]; C statistic, 0.55 [95% CI, 0.48-0.62]). Discriminative accuracy was higher when physicians and nurses were confident about their predictions (eg, for physicians' confident predictions of 6-month mortality: positive LR, 33.00 [95% CI, 8.34-130.63]; negative LR, 0.18 [95% CI, 0.09-0.35]; C statistic, 0.90 [95% CI, 0.84-0.96]). Compared with a predictive model including objective clinical variables, a model that also included physician and nurse predictions had significantly higher discriminative accuracy for in-hospital mortality, 6-month mortality, and return to original residence (P < .01 for all).

Conclusions and relevance: ICU physicians' and nurses' discriminative accuracy in predicting 6-month outcomes of critically ill patients varied depending on the outcome being predicted and confidence of the predictors. Further research is needed to better understand how clinicians derive prognostic estimates of long-term outcomes.

Figure.. Distribution of Self-Reported Confidence in Predictions for Physicians and Nurses^a

^aReported confidence in all predictions. For unable to toilet independently at 6 months, unable to ambulate up 10 stairs at 6 months, and abnormal cognition at 6 months, this includes the confidence in all predictions, not just among patients who survived. For ranking of confidence, 1 indicates “not confident at all”; 2, “slightly confident”; 3, “moderately confident”; 4, “considerably confident”; and 5, “very confident.”