External validation and comparison of six cardiovascular risk prediction models in the Prospective Urban Rural Epidemiology (PURE)-Colombia study

Abstract

Aims: To externally validate the SCORE2, AHA/ACC pooled cohort equation (PCE), Framingham Risk Score (FRS), Non-Laboratory INTERHEART Risk Score (NL-IHRS), Globorisk-LAC, and WHO prediction models and compare their discrimination and calibration capacity.

Methods and results: Validation in individuals aged 40-69 years with at least 10 years of follow-up and without baseline use of statins or cardiovascular diseases from the Prospective Urban Rural Epidemiology (PURE)-Colombia prospective cohort study. For discrimination, the C-statistic, and receiver operating characteristic curves with the integrated area under the curve (AUCi) were used and compared. For calibration, the smoothed time-to-event method was used, choosing a recalibration factor based on the integrated calibration index (ICI). In the NL-IHRS, linear regressions were used. In 3802 participants (59.1% women), baseline risk ranged from 4.8% (SCORE2 women) to 55.7% (NL-IHRS). After a mean follow-up of 13.2 years, 234 events were reported (4.8 cases per 1000 person-years). The C-statistic ranged between 0.637 (0.601-0.672) in NL-IHRS and 0.767 (0.657-0.877) in AHA/ACC PCE. Discrimination was similar between AUCi. In women, higher over-prediction was observed in the Globorisk-LAC (61%) and WHO (59%). In men, higher over-prediction was observed in FRS (72%) and AHA/ACC PCE (71%). Overestimations were corrected after multiplying by a factor derived from the ICI.

Conclusion: Six prediction models had a similar discrimination capacity, supporting their use after multiplying by a correction factor. If blood tests are unavailable, NL-IHRS is a reasonable option. Our results suggest that these models could be used in other countries of Latin America after correcting the overestimations with a multiplying factor.

Keywords: Cardiovascular disease; Latin-American; Prediction; Primary prevention; Scores; Validation.

Figure 1

Time-dependent area under the receiver operating characteristic curve for cardiovascular risk prediction models. The diagonal line represents perfect chance (with a corresponding area under the receiver operating characteristic curve of 0.5), values may range to 1.0 (predict prediction).

Figure 2

Calibration of the six prediction models for women and men. Plots depict mean observed risk and predicted 10-year risk across deciles of predicted risk. Values below the 1 slope represent overestimation, while values above the 1 slope represent underestimation.

Figure 3

Calibration and recalibration plots of predicted and observed 10-year cardiovascular risk in the prediction models. The figure shows the relationship between observed/estimated events in the initial calibration (Calibrated) and after recalibration with the integrated calibration index for each model (Recalibrated).