External Validation and Recalibration of Risk Prediction Models for Acute Traumatic Brain Injury among Critically Ill Adult Patients in the United Kingdom

Abstract

This study validates risk prediction models for acute traumatic brain injury (TBI) in critical care units in the United Kingdom and recalibrates the models to this population. The Risk Adjustment In Neurocritical care (RAIN) Study was a prospective, observational cohort study in 67 adult critical care units. Adult patients admitted to critical care following acute TBI with a last pre-sedation Glasgow Coma Scale score of less than 15 were recruited. The primary outcomes were mortality and unfavorable outcome (death or severe disability, assessed using the Extended Glasgow Outcome Scale) at six months following TBI. Of 3626 critical care unit admissions, 2975 were analyzed. Following imputation of missing outcomes, mortality at six months was 25.7% and unfavorable outcome 57.4%. Ten risk prediction models were validated from Hukkelhoven and colleagues, the Medical Research Council (MRC) Corticosteroid Randomisation After Significant Head Injury (CRASH) Trial Collaborators, and the International Mission for Prognosis and Analysis of Clinical Trials in TBI (IMPACT) group. The model with the best discrimination was the IMPACT "Lab" model (C index, 0.779 for mortality and 0.713 for unfavorable outcome). This model was well calibrated for mortality at six months but substantially under-predicted the risk of unfavorable outcome. Recalibration of the models resulted in small improvements in discrimination and excellent calibration for all models. The risk prediction models demonstrated sufficient statistical performance to support their use in research and audit but fell below the level required to guide individual patient decision-making. The published models for unfavorable outcome at six months had poor calibration in the UK critical care setting and the models recalibrated to this setting should be used in future research.

Keywords: outcome measures; prospective study; traumatic brain injury.

FIG. 1.

Calibration plots—mortality at six months (published models) Observed mortality is plotted against predicted mortality in 10 equal sized groups. The gray dashed line is from a locally-weighted sum of squares smoother. The distribution of the predicted probability of mortality is shown at the foot of each figure. Calibration plots are shown for all patients; results were similar when restricted to patients eligible for each model.

FIG. 2.

Calibration plots—unfavorable outcome at six months (published models) Observed unfavorable outcome is plotted against predicted unfavorable outcome in 10 equal sized groups. The gray dashed line is from a locally-weighted sum of squares smoother. The distribution of the predicted probability of unfavorable outcome is shown at the foot of each figure. Calibration plots are shown for all patients; results were similar when restricted to patients eligible for each model.

FIG. 3.

Calibration plots—mortality at six months (recalibrated models) Observed mortality is plotted against predicted mortality in 10 equal sized groups. The gray dashed line is from a locally-weighted sum of squares smoother. The distribution of the predicted probability of mortality is shown at the foot of each figure.

FIG. 4.

Calibration plots—unfavorable outcome at six months (recalibrated models) Observed unfavorable outcome is plotted against predicted unfavorable outcome in 10 equal sized groups. The gray dashed line is from a locally-weighted sum of squares smoother. The distribution of the predicted probability of unfavorable outcome is shown at the foot of each figure.