Clinician Accuracy in Identifying and Predicting Organ Dysfunction in Critically Ill Children

Abstract

Objectives: To determine clinician accuracy in the identification and prediction of multiple organ dysfunction syndrome.

Design: Prospective cohort study.

Setting: University of Michigan's C.S. Mott Children's Hospital PICU.

Patients: Patients admitted to the PICU with an anticipated PICU length of stay greater than 48 hours.

Interventions: None.

Measurements and main results: For each patient, the clinical team (attending, fellow, resident/nurse practitioner) was surveyed regarding existing and anticipated organ dysfunction. The primary outcomes were clinicians' accuracy at identifying multiple organ dysfunction syndrome and predicting new or progressive multiple organ dysfunction syndrome, compared to the objective assessment of multiple organ dysfunction syndrome using Proulx criteria. We also measured sensitivity, specificity, negative and positive predictive values, and negative and positive likelihood ratios of clinician assessments. We tested for differences in accuracy by clinician type using chi-square tests. Clinicians rated their confidence in prediction on a 5-point Likert scale. There were 476 eligible PICU admissions, for whom 1,218 surveys were completed. Multiple organ dysfunction syndrome was present in 89 patients (18.7%) at enrollment, and new or progressive multiple organ dysfunction syndrome occurred in 39 (8.2%). Clinicians correctly identified multiple organ dysfunction syndrome with 79.9% accuracy and predicted additional organ dysfunction with 82.6% accuracy. However, the positive and negative likelihood ratios for new or progressive multiple organ dysfunction syndrome prediction were 3.0 and 0.7, respectively, indicating a weak relationship between the clinician prediction and development of new or progressive multiple organ dysfunction syndrome. The positive predictive value of new or progressive multiple organ dysfunction syndrome prediction was just 22.1%. We found no differences in accuracy by clinician type for either identification of multiple organ dysfunction syndrome (80.2% vs 78.2% vs 81.0%; p = 0.57) or prediction of new or progressive multiple organ dysfunction syndrome (84.8% vs 82.8% vs 80.3%; p = 0.26) for attendings, fellows, and residents/nurse practitioners, respectively. There was a weak correlation between the confidence and accuracy of prediction (pairwise correlation coefficient, 0.26; p < 0.001).

Conclusions: PICU clinicians correctly identified multiple organ dysfunction syndrome and predicted new or progressive multiple organ dysfunction syndrome with 80% accuracy. However, only 8% of patients developed new or progressive multiple organ dysfunction syndrome, so accuracy was largely due to true negative predictions. The positive predictive value for new or progressive multiple organ dysfunction syndrome prediction was just 22%. Accuracy did not differ by clinician type, but was correlated with self-rated confidence and was higher for negative predictions.

Figure 1:

Flowchart of Enrollment and New or Progressive Multiple Organ Dysfunction Syndrome

Figure 2:. Clinician Identification and Prediction of Organ Dysfunction on Enrollment and New Organ Dysfunctions

These stacked bar graphs represent the percentage of survey responses for identification and prediction of Multiple Organ Dysfunction Syndrome (MODS) and New or Progressive MODS (NP-MODS) at baseline and by day 7 of pediatric intensive care unit admission. Each individual organ system is also presented. The yellow bars represent those patients with organ dysfunction, while blue bars represent those without.

Figure 3:. Self-Reported Confidence in Prediction of Organ Dysfunction by Clinicians

For certainty ratings, 1 indicates not very certain, while 5 indicates very certain that the patient will or will not develop organ dysfunction within the next 7 days.