Machine-learning-based prediction of disability progression in multiple sclerosis: An observational, international, multi-center study

Abstract

Background: Disability progression is a key milestone in the disease evolution of people with multiple sclerosis (PwMS). Prediction models of the probability of disability progression have not yet reached the level of trust needed to be adopted in the clinic. A common benchmark to assess model development in multiple sclerosis is also currently lacking.

Methods: Data of adult PwMS with a follow-up of at least three years from 146 MS centers, spread over 40 countries and collected by the MSBase consortium was used. With basic inclusion criteria for quality requirements, it represents a total of 15, 240 PwMS. External validation was performed and repeated five times to assess the significance of the results. Transparent Reporting for Individual Prognosis Or Diagnosis (TRIPOD) guidelines were followed. Confirmed disability progression after two years was predicted, with a confirmation window of six months. Only routinely collected variables were used such as the expanded disability status scale, treatment, relapse information, and MS course. To learn the probability of disability progression, state-of-the-art machine learning models were investigated. The discrimination performance of the models is evaluated with the area under the receiver operator curve (ROC-AUC) and under the precision recall curve (AUC-PR), and their calibration via the Brier score and the expected calibration error. All our preprocessing and model code are available at https://gitlab.com/edebrouwer/ms_benchmark, making this task an ideal benchmark for predicting disability progression in MS.

Findings: Machine learning models achieved a ROC-AUC of 0⋅71 ± 0⋅01, an AUC-PR of 0⋅26 ± 0⋅02, a Brier score of 0⋅1 ± 0⋅01 and an expected calibration error of 0⋅07 ± 0⋅04. The history of disability progression was identified as being more predictive for future disability progression than the treatment or relapses history.

Conclusions: Good discrimination and calibration performance on an external validation set is achieved, using only routinely collected variables. This suggests machine-learning models can reliably inform clinicians about the future occurrence of progression and are mature for a clinical impact study.

Fig 1. Overall layout of our approach.

A: Representation of a clinical trajectory of an individual person with multiple sclerosis (PwMS). The trajectory consists of, among others, relapses, EDSS values, and treatment durations collected over time. The full list of used variables is given in the Materials and Methods. The trajectory of each patient is divided into an observation window (the available clinical history for the prediction) and the future trajectory, which is used to compute the confirmed disability progression label at two years (w_c). B: For an individual PwMS, the clinical trajectory in the observation window is extracted and used in the machine learning model to predict a well-calibrated probability of disability progression at two years. Based on the predictions, clinicians can adjust their clinical decisions accordingly. C: The MSbase dataset contains clinical data from 146 individual MS clinical centers with different clinical practice. We leveraged this feature by creating an external validation cohort of patients. We split the data per clinic, with 60% of patients used for training the model, 20% for optimizing the hyper-parameters (validation set) and 20% for external validation. The results presented in this work are all on the external validation cohort.

Fig 2. Visual representation of the discrimination performance.

ROC-AUC curve, the AUC-PR curve, and distribution of the estimated probability of relapse per group obtained with the temporal attention model.

Fig 3. Calibration diagram for the temporal attention model for the first data split.

The val.prob.ci.2 function [18] was used to generate this plot.

Fig 4. Feature importance of different variables.

Feature importance of different variables used in the MLP model based on the average performance degradation on the ROC-AUC, AUC-PR, and ECE metrics. ‘EDSS at 0’ stands for the Expanded Disability Status Scale score at the time of prediction. ‘Date Reference’ represents the date of prediction. ‘Mean EDSS’ stands for the average EDSS over the last 3 years. ‘MS Course = SP’ is a binary variable indicating that the MS course is secondary progressive at the time of prediction. ‘Mean KFS x’ represents the corresponding variable in the average Kurtzke Functional Systems Score over the last three years. ‘Std EDSS’ represents the standard deviation of EDSS over the last 3 years.

Fig 5. Flowchart of patient selection.

Flowchart of patient selection for both at least three and at least six visits in the last 3.25 years.

Fig 6. Problem Setup.

A: For each patient episode, the available data for prediction consists of the baseline data and the longitudinal clinical data in the observation window. Disability progression (w_c) was assessed based on the difference between the EDSS at time t = 0 and two years later (t = t_2y) as defined in Eq (1). B: Based on the available historical clinical data (in the observation time window), we aimed at training a model able to predict the probability p(w_c) of disability progression at a two years horizon (t_2y).

Fig 7. Examples of valid and non-valid episodes.

The time is in years (y) and months (m). (a) Confirmed progression after two years. The EDSS around 2y6m is not used to confirm the progression, because it occurs within 1 month after a relapse. Progression is confirmed with the EDSS measurement around 4y. There are 3 EDSS measurements between −3y and 0y, which is enough follow-up data. (b) This is not a valid sample: there are not enough EDSS measurements between −3y and 0y. (c) This is not a valid sample: no confirmed progression because there are no EDSS values after 2y. (d) This is a valid sample: the EDSS decreases after 2y, so this counts as no disability progression. (e) This is a valid sample: w_u = 0, so no confirmation is needed.

Fig 8. TRIPOD checklist.