Value of electronic data for model validation and refinement: bacteremia risk in children with fever and neutropenia

Abstract

Background: Validating published risk models in a different time and setting can be a labor-intensive process. Data in electronic format provide the potential to test the validity of risk models without labor-intensive chart reviews and data capture. The authors attempted to use readily available electronic data to find appropriate cases and to validate and refine a previously developed risk model for predicting bacteremia in children with cancer who had fever and neutropenia.

Patients and methods: By applying a largely automated case-finding algorithm to linked, electronic clinical and administrative data systems, the authors identified and acquired data regarding 157 episodes of fever and neutropenia in children with cancer admitted to a children's hospital during an 11-month period in 1997. The authors applied a previously developed and validated risk model for bacteremia to this 1997 cohort by assessing the odds ratios among risk groups. The model assigns encounters with absolute monocyte count of 100 cells or more/mm3 to a low-risk group and encounters with an absolute monocyte count of less than 100 cells/mm3 to intermediate-risk (temperature <39.0 degrees C) or high-risk (> or = 39.0 degrees C) groups. In addition, the authors explored whether the new data would have generated the same model. Univariate and multivariable analyses were performed to determine whether there were additional independent predictors of bacteremia. Recursive partitioning of admission absolute monocyte count and temperature was used to assess whether similar cutpoints would be found.

Results: There were 12 episodes of bacteremia (7.6%) among the 157 encounters. The previously developed model correctly predicted increasing rates of bacteremia in this 1997 cohort, ranging from 2.5% in the low-risk group (one episode in a child with an infected central line) to 24% in the high-risk group. The odds ratio for the high-risk versus intermediate-risk group was 4.09 (95% confidence interval 1.05-15.91), comparable to the odds ratio of 3.96 in the previously published derivation cohort (95% confidence interval 1.4-11.1). Multivariate analysis of the new data revealed no independent risk factors for bacteremia other than admission absolute monocyte count and temperature. Recursive partitioning of absolute monocyte count and temperature generated risk categories that were somewhat different from those of the original model. The new data yielded three categories: low risk (temperature < or = 39.5 degrees C and absolute monocyte count >10/mm3), intermediate risk (temperature < or = 39.5 degrees C and absolute monocyte count < or = 10/mm3), and high risk (temperature >39.5 degrees C).

Conclusions: Existing electronic data provide an efficient means for case-finding and model validation and refinement. The previously developed bacteremia model had good but not optimal predictive performance in the new data set. Admission absolute monocyte count and temperature remain significant risk factors for bacteremia. Redefining the risk categories, including a much lower cutpoint for admission absolute monocyte count, improved the model's discrimination, which suggests that predictive models need periodic updating.