Validation of Childhood Pneumonia Prognostic Models for Use in Emergency Care Settings

Abstract

Background: Unwarranted variation in disposition decisions exist among children with pneumonia. We validated three prognostic models for predicting pneumonia severity among children in the emergency department (ED) and hospital.

Methods: We performed a two-center, prospective study of children 6 months to <18 years presenting to the ED with pneumonia from January 2014 to May 2019. We evaluated three previously developed disease-specific prognostic models which use demographic, clinical, and diagnostic predictor variables, with each model estimating risk for Very Severe (mechanical ventilation or shock), Severe (ICU without very severe features), and Moderate/Mild (Hospitalization without severe features or ED discharge) pneumonia. Predictive accuracy was measured using discrimination (concordance or c-statistic) and re-calibration.

Results: There were 1088 children included in one or more of the three models. Median age was 3.6 years and the majority of children were male (53.7%) and identified as non-Hispanic White (63.7%). The distribution for the ordinal severity outcome was mild or moderate (79.1%), severe (15.9%), and very severe (4.9%). The three models each demonstrated excellent discrimination (C-statistic range across models [0.786-0.803]) with no appreciable degradation in predictive accuracy from the derivation cohort.

Conclusions: All three prognostic models accurately identified risk for three clinically meaningful levels of pneumonia severity and demonstrated very good predictive performance. Physiologic variables contributed the most to model prediction. Application of these objective tools may help standardize and improve disposition and other management decisions for children with pneumonia.

Keywords: pediatrics; pneumonia; prognostic model; validation.

Figure 1.

Flow diagram. For the final models, outcomes are represented in a horizontal stacked bar graph where black indicates mild (ED discharge), light gray—moderate (inpatient floor), dark gray—severe (ICU without mechanical ventilation, shock or death), white—very severe (ICU with mechanical ventilation, shock or death). The number within the stacked bar graph indicates proportion of individuals with each outcome.

Figure 2.

Calibration plots and predicted probabilities pediatric pneumonia prognostic models. Calibration plots of the Full, Expert and EHR models after re-calibration. The solid line represents a nonparametric smooth curve between observed proportion and predicted probability. The solid line represents a nonparametric smooth curve between observed proportion and predicted probability. Perfect calibration is represented by the dotted line through the origin. Triangles are based on deciles of patients grouped by similar predicted probabilities. The distribution of subjects is indicated with spikes at the top (for subjects with severe/very severe outcome) and the bottom (for subjects with mild/moderate outcome) of the graph. Maximum and mean absolute error (Harrell’s E-statistic) are also presented. The estimates in terms of intercepts for severe outcome were −0.11(95% CI, −0.46 to 0.24), −0.36 (95% CI, −0.67 to −0.05) and −0.52 (95% CI, −0.84 to −0.2). The calibration slopes were estimated to be 0.85 (p = 0.05), 0.94 (p = 0.36) and 0.96 (p = 0.66) and none of them significantly deviated from ideal calibration line.

Figure 3.

Receiver-operator curves of the prediction models distinguishing between mild/moderate and severe/very severe outcomes. The cutoff (the predicted probability of being severe/very severe) was chosen to maximize the sum of sensitivity and specificity.