Determinants of the calibration of SAPS II and SAPS 3 mortality scores in intensive care: a European multicenter study

Abstract

Background: The aim of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 is to predict the mortality of patients admitted to intensive care units (ICUs). Previous studies have suggested that the calibration of these scores may vary across countries, centers, and/or characteristics of patients. In the present study, we aimed to assess determinants of the calibration of these scores.

Methods: We assessed the calibration of the SAPS II and SAPS 3 scores among 5266 patients admitted to ICUs during a 4-week period at 120 centers in 17 European countries. We obtained calibration curves, Brier scores, and standardized mortality ratios. Points attributed to SAPS items were reevaluated and compared with those of the original scores. Finally, we tested associations between the calibration and center characteristics.

Results: The mortality was overestimated by both scores: The standardized mortality ratios were 0.75 (95% CI 0.71-0.79) for the SAPS II score and 0.91 (95% CI 0.86-0.96) for the SAPS 3 score. This overestimation was partially explained by changes in associations between some items of the scores and mortality, especially the heart rate, Glasgow Coma Scale score, and diagnosis of AIDS for SAPS II. The calibration of both scores was better in countries with low health expenditures. The between-center variability in calibration curves was much greater than expected by chance.

Conclusions: Both scores overestimate current mortality among European ICU patients. The magnitude of the miscalibration of SAPS II and SAPS 3 scores depends not only on patient characteristics but also on center characteristics. Furthermore, much between-center variability in calibration remains unexplained by these factors.

Trial registration: ClinicalTrials.gov identifier: NCT01422070 . Registered 19 August 2011.

Keywords: Calibration; Determinants; SAPS 3; SAPS II.

Fig. 1

Calibration curves for the Simplified Acute Physiology Score (SAPS) II (dark line) and the SAPS 3 (gray line) obtained with a kernel function. The calibration curve represents the relationship between the mortality predicted by the score (x-axis) and the observed mortality (y-axis). The identity line (dashed line) represents a perfect calibration. A calibration curve below the identity line indicates that the score overestimates the mortality. The black and gray circles represent the estimates of the observed mortality in sample, stratified by levels of predicted mortality (by step of 0.01 up to a predicted mortality of 0.20, by step of 0.025 for a predicted mortality from 0.20 to 0.35, and by step of 0.05 for a predicted mortality greater than 0.35). The size of the circles is proportional to the number of patients in categories of predicted mortality

Fig. 2

Calibration curves of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 obtained by kernel function by reason for admission to the intensive care unit. a Basic observation. b Severe trauma. c Respiratory reason. d Cardiovascular reason. e Digestive reason. f Neurological reason

Fig. 3

Observed and expected calibration curves for the Simplified Acute Physiology Score (SAPS) II score (top) and SAPS 3 score (bottom) in 120 centers and between-center variability in standardized mortality ratio (SMR). a Calibration curves of SAPS II in each of the 120 centers fitted with a logistic regression model. The black line represents the overall calibration curve. b Expected calibration curves of SAPS II under the assumption that the calibration is the same in all centers. The represented between-center variability is the random (sampling) variability. c Distribution of the SD of the center-specific SMRs under the assumption that the calibration of SAPS II is the same in all centers. The vertical line represents the observed value of the SD. d–f are the same figures shown in a–c repeated for the SAPS 3 score

Fig. 4

Calibration curves of the Simplified Acute Physiology Score (SAPS) II score obtained by kernel function according to (a) health expenditure expressed in percentage of gross domestic product (GDP) and (b) number of hospital beds, and calibration curves of the SAPS 3 score by (c) health expenditure expressed as a percentage of GDP and (d) daytime nurse/patient ratio