Temporal validation of 30-day mortality prediction models for transcatheter aortic valve implantation using statistical process control - An observational study in a national population

Abstract

Background: Various mortality prediction models for Transcatheter Aortic Valve Implantation (TAVI) have been developed in the past years. The effect of time on the performance of such models, however, is unclear given the improvements in the procedure and changes in patient selection, potentially jeopardizing the usefulness of the prediction models in clinical practice. We aim to explore how time affects the performance and stability of different types of prediction models of 30-day mortality after TAVI.

Methods: We developed both parametric (Logistic Regression) and non-parametric (XGBoost) models to predict 30-day mortality after TAVI using data from the Netherlands Heart Registration. The models were trained with data from 2013 to the beginning of 2016 and pre-control charts from Statistical Process Control were used to analyse how time affects the models' performance on independent data from the mid of 2016 to the end of 2019. The area under the Receiver Operating Characteristics curve (AUC) was used to evaluate the models in terms of discrimination and the Brier Score (BS), which is related to calibration, in terms of accuracy of the predicted probabilities. To understand the extent to which refitting the models contribute to the models' stability, we also allowed the models to be updated over time.

Results: We included data from 11,291 consecutive TAVI patients from hospitals in the Netherlands. The parametric model without re-training had a median AUC of 0.64 (IQR 0.54-0.73) and BS of 0.028 (IQR 0.021-0.035). For the non-parametric model, the median AUC was 0.63 (IQR 0.48-0.68) and BS was 0.027 (IQR 0.021-0.036). Over time, the developed parametric model was stable in terms of AUC and unstable in terms of BS. The non-parametric model was considered unstable in both AUC and BS. Repeated model refitting resulted in stable models in terms of AUC and decreased the variability of BS, although BS was still unstable. The refitted parametric model had a median AUC of 0.66 (IQR 0.57-0.73) and BS of 0.027 (IQR 0.020-0.035) while the non-parametric model had a median AUC of 0.66 (IQR 0.57-0.74) and BS of 0.027 (IQR 0.023-0.035).

Conclusions: The temporal validation of the TAVI 30-day mortality prediction models showed that the models refitted over time are more stable and accurate when compared to the frozen models. This highlights the importance of repeatedly refitted models over time to improve or at least maintain their performance stability. The non-parametric approach did not show improvement over the parametric approach.

Keywords: Aortic stenosis; Machine learning; Prediction models; Statistical process control; Temporal validation; Transcatheter aortic valve implantation.

Fig. 1

Schematic representation of the experiments with a frozen model and model refitting scenario. The frozen model was kept unchanged for all iterations while model refitting implied re-training the model in every new iteration.

Fig. 2

Mean 30-day mortality and age over time of TAVI patients. A linear trend is presented in red. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 3

Temporal validation of the frozen LR (left) and XGB (right) models. For the LR model, the AUC is considered stable and most of the BS points are inside the red zone, hence the BS is unstable. Regarding the XGB model, an AUC point and most of the BS points are inside the red zone, hence the process is unstable. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 4

Temporal validation of the model refitting for LR (left) and XGB (right). While the AUC is stable for both models, some of the BS points are in the red zone at the beginning, hence the BS is unstable. Note that the zone limits are recalculated per model refitting on a successive group. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 5

Calibration plots of the LR and XGB models. The plots were generated after the combination of all data points.