Machine learning-based prediction of 1-year all-cause mortality in patients undergoing CRT implantation: validation of the SEMMELWEIS-CRT score in the European CRT Survey I dataset

Abstract

Aims: We aimed to externally validate the SEMMELWEIS-CRT score for predicting 1-year all-cause mortality in the European Cardiac Resynchronization Therapy (CRT) Survey I dataset-a large multi-centre cohort of patients undergoing CRT implantation.

Methods and results: The SEMMELWEIS-CRT score is a machine learning-based tool trained for predicting all-cause mortality in patients undergoing CRT implantation. This tool demonstrated impressive performance during internal validation but has not yet been validated externally. To this end, we applied it to the data of 1367 patients from the European CRT Survey I dataset. The SEMMELWEIS-CRT predicted 1-year mortality with an area under the receiver operating characteristic curve (AUC) of 0.729 (0.682-0.776), which concurred with the performance measured during internal validation [AUC: 0.768 (0.674-0.861), P = 0.466]. Moreover, the SEMMELWEIS-CRT score outperformed multiple conventional statistics-based risk scores, and we demonstrated that a higher predicted probability is not only associated with a higher risk of death [odds ratio (OR): 1.081 (1.061-1.101), P < 0.001] but also with an increased risk of hospitalizations for any cause [OR: 1.013 (1.002-1.025), P = 0.020] or for heart failure [OR: 1.033 (1.015-1.052), P < 0.001], a less than 5% improvement in left ventricular ejection fraction [OR: 1.033 (1.021-1.047), P < 0.001], and lack of improvement in New York Heart Association functional class compared with baseline [OR: 1.018 (1.006-1.029), P = 0.003].

Conclusion: In the European CRT Survey I dataset, the SEMMELWEIS-CRT score predicted 1-year all-cause mortality with good discriminatory power, which confirms the generalizability and demonstrates the potential clinical utility of this machine learning-based risk stratification tool.

Keywords: All-cause death; Cardiac resynchronization therapy; Heart failure; Machine learning; Risk stratification.

Structured Graphical Abstract

Figure 1

Patient selection flowchart. CRT, cardiac resynchronization therapy; eCRF, electronic case report form; ICD, implantable cardioverter-defibrillator.

Figure 2

Discriminatory power of the SEMMELWEIS-CRT score in the internal validation cohort and the European CRT Survey I dataset. Receiver operating characteristic curves are plotted with 95% confidence bands. Areas under the receiver operating characteristic curve are presented with 95% confidence intervals. AUC, area under the receiver operating characteristic curve.

Figure 3

Performance of the SEMMELWEIS-CRT score vs. other conventional statistics-based risk scores (A) and the associations between the SEMMELWEIS-CRT-predicted probability and unfavourable outcomes (B). *P < 0.05 vs. SEMMELWEIS-CRT, DeLong test. Odds ratios with 95% confidence intervals are calculated for a 0.01 increase in the predicted probability. AUC, area under the receiver operating characteristic curve; CI, confidence interval; HF, heart failure; NYHA, New York Heart Association; LVEF, left ventricular ejection fraction; OR, odds ratio.

Figure 4

Contribution of each input feature to the performance of the SEMMELWEIS-CRT score in predicting 1-year mortality in the European CRT Survey I dataset. To identify the input features contributing the most to the SEMMELWEIS-CRT score’s performance in the analysed subset of the European CRT Survey I dataset, we computed permutation feature importance. This technique measures the importance of each input feature by calculating the decrease in the model’s performance (i.e. area under the receiver operating characteristic curve) after randomly shuffling the feature’s values (while keeping the other features the same as before). A feature is considered important if shuffling its values results in a substantial decrease in the model’s performance. In the current study, we performed permutation 20 times for each feature. The final order of importance was determined based on the median decrease in area under the receiver operating characteristic curve (marked by a vertical line in each box). ACE-I, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; AUC, area under the receiver operating characteristic curve; CRT-D, cardiac resynchronization therapy defibrillator; CRT-P, cardiac resynchronization therapy pacemaker; GFR, glomerular filtration rate; HF, heart failure; LVEF, left ventricular ejection fraction; MRA, mineralocorticoid receptor antagonist; NT-proBNP, N-terminal pro-brain natriuretic peptide; NYHA, New York Heart Association; SBP, systolic blood pressure.