Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings

Abstract

In patients with community-acquired pneumonia (CAP) prediction rules based on individual predicted mortalities are frequently used to support decision-making for in-patient vs. outpatient management. We studied the accuracy and the need for recalibration of three risk prediction scores in a tertiary-care University hospital emergency-department setting in Switzerland. We pooled data from patients with CAP enrolled in two randomized controlled trials. We compared expected mortality from the original pneumonia severity index (PSI), CURB65 and CRB65 scores against observed mortality (calibration) and recalibrated the scores by fitting the intercept alpha and the calibration slope beta from our calibration model. Each of the original models underestimated the observed 30-day mortality of 11%, in 371 patients admitted to the emergency department with CAP (8.4%, 5.5% and 5.0% for the PSI, CURB65 and CRB65 scores, respectively). In particular, we observed a relevant mortality within the low risk classes of the original models (2.6%, 5.3%, and 3.7% for PSI classes I-III, CURB65 classes 0-1, and CRB65 class 0, respectively). Recalibration of the original risk models corrected the miscalibration. After recalibration, however, only PSI class I was sensitive enough to identify patients with a low risk (i.e. <1%) for mortality suitable for outpatient management. In our tertiary-care setting with mostly referred in-patients, CAP risk scores substantially underestimated observed mortalities misclassifying patients with relevant risks of death suitable for outpatient management. Prior to the implementation of CAP risk scores in the clinical setting, the need for recalibration and the accuracy of low-risk re-classification should be studied in order to adhere with discharge guidelines and guarantee patients' safety.

Fig. 1

Flow chart of all 373 patients included in this study. CAP, Community-acquired pneumonia.

Fig. 2

Agreement between predicted and observed 30-day mortality (calibration) for three pneumonia severity prediction rules (a) PSI, (b) CURB65 and (c) CRB65. Observed mortality is plotted according to classes of predicted risk for each prediction rule separately. The solid line of identity represents perfect calibration of predicted risk within new patients. Correction of miscalibration () after recalibration (◊).

Fig. 3

Receiver-operating characteristics analysis for 30-day mortality prediction with three original pneumonia severity prediction rules (PSI, CURB65 and CRB65) in 373 patients with CAP.