Predicting outcome after traumatic brain injury: development of prognostic scores based on the IMPACT and the APACHE II

Abstract

Prediction models are important tools for heterogeneity adjustment in clinical trials and for the evaluation of quality of delivered care to patients with traumatic brain injury (TBI). We sought to improve the predictive performance of the IMPACT (International Mission for Prognosis and Analysis of Clinical Trials) prognostic model by combining it with the APACHE II (Acute Physiology and Chronic Health Evaluation II) for 6-month outcome prediction in patients with TBI treated in the intensive care unit. A total of 890 patients with TBI admitted to a large urban level 1 trauma center in 2009-2012 comprised the study population. The IMPACT and the APACHE II scores were combined using binary logistic regression. A randomized, split-sample technique with secondary bootstrapping was used for model development and internal validation. Model performance was assessed by discrimination (by area under the curve [AUC]), calibration, precision, and net reclassification improvement (NRI). Overall 6-month mortality was 22% and unfavorable neurological outcome 47%. The predictive power of the new combined IMPACT-APACHE II models was significantly superior, compared to the original IMPACT models (AUC, 0.81-0.82 vs. 0.84-0.85; p<0.05) for 6-month mortality prediction, but not for unfavorable outcome prediction (AUC, 0.81-0.82 vs. 0.83; p>0.05). However, NRI showed a significant improvement in risk stratification of patients with unfavorable outcome by the IMPACT-APACHE II models, compared to the original models (NRI, 5.4-23.2%; p<0.05). Internal validation using split-sample and resample bootstrap techniques yielded equivalent results, indicating low grade of overestimation. Our findings show that by combining the APACHE II with the IMPACT, improved 6-month outcome predictive performance is achieved. This may be applicable for heterogeneity adjustment in forthcoming TBI studies.

Keywords: APACHE II; IMPACT; external validation; outcome; prognostic models; traumatic brain injury.

FIG. 1.

Study population. ICD, International Statistical Classification of Diseases; GOS, Glasgow Outcome Scale.

FIG. 2.

Calibration tests for the original IMPACT and APACHE II models for 6-month mortality (left) and unfavorable outcome (right) prediction. The H-L calibration plots to the left (with a loess smoother curve fitted between the groups) and the GiViTI calibration belts to the right for 6-month mortality and neurological outcome, respectively. The GiViTI belt shows risk intervals of significant under- and overprediction when the 95% confidence interval does not encompass the diagonal bisector line (black line, indicating perfect calibration). The APACHE II model showed good calibration for mortality (p=0.099), but not neurological outcome (p<0.001) prediction. Both the IMPACT core and extended models showed poor calibration by both tests (p<0.05). The IMPACT lab was the only model showing good calibration for both mortality (p=0.054) and neurological outcome (p=0.078) prediction. IMPACT, International Mission for Prognosis and Analysis of Clinical Trials; APACHE II, Acute Physiology and Chronic Health Evaluation II; H-L, the Hosmer-Lemeshow Ĉ-test; GiViTI, Italian Group for the Evaluation of Interventions in Intensive Care Medicine.

FIG. 3.

Area under the receiver operator characteristic curve (AUC) for mortality (top) and neurological outcome (bottom) prediction. The IMPACTcore–APACHE II (left), the IMPACText–APACHE II (middle), and the IMPACTlab-APACHE II (right). The AUCs are compared between the models with a concomitant p value (p<0.05 indicates a significant difference). All new models showed significantly higher AUCs, compared to the original IMPACT and APACHE II models, for mortality prediction. The new models showed significantly higher AUCs, compared to the APACHE II, but not compared to the original IMPACT models, for neurological outcome prediction. IMPACT, International Mission for Prognosis and Analysis of Clinical Trials; APACHE II, Acute Physiology and Chronic Health Evaluation II; CI, confidence interval.

FIG. 4.

Calibration tests for the newly developed IMPACT–APACHE II models in the validation cohort. Calibration for mortality prediction (left) and for neurological outcome prediction (right). The H-L calibration plots (left; with a loess smoother curve fitted between the groups) and the GiViTI calibration belts (right) for mortality and neurological outcome, respectively. All new models showed good calibration by the H-L test (p>0.05). Only the IMPACText–APACHE II showed poor calibration by the H-L test (p=0.037). Accordingly, the GiViTI calibration belt reveals significant under prediction (95% confidence interval over the diagonal bisector line) between for a risk interval between 0.61 and 0.97. IMPACT, International Mission for Prognosis and Analysis of Clinical Trials; APACHE II, Acute Physiology and Chronic Health Evaluation II; H-L, the Hosmer-Lemeshow Ĉ-test; GiViTI, Italian Group for the Evaluation of Interventions in Intensive Care Medicine.