Deep learning to predict progression independent of relapse activity at a first demyelinating event

Abstract

Progression independent of relapse activity is the main cause of irreversible disability in multiple sclerosis and is strongly associated with older age at symptom onset. Early and accurate prediction, at symptom onset, of which patients are at highest risk of progression independent of relapses, is an unmet need. This study aimed to develop a deep learning survival model using only routine MRI acquired at the first demyelinating attack to predict the risk of progression independent of relapses, and assess its ability to improve classical age-adjusted predictions. We analysed a prospective cohort of patients under 50, clinically assessed within three months of symptom onset, with available MRI (T1- and T2-Fluid-Attenuated Inversion Recovery sequences). An independent early multiple sclerosis cohort (≤1 year from symptom onset) from the Multiple Sclerosis Partners Advancing Technology and Health Solutions database (N = 32) was used for external validation. Patients were assessed for progression independent of relapse activity, defined as a 6-month confirmed increase in the Expanded Disability Status Scale without relapses. Our deep learning model used EfficientNet to estimate the cumulative probability of progression independent of relapses at 1-year intervals. We employed 5-fold cross-validation for model training and testing, assessing performance with the time-dependent concordance index. We also investigated the optimal cumulative probability threshold for binary risk stratification. The model's ability to improve a classical Cox regression model was evaluated. Additionally, we identified brain regions most relevant to deep learning-based progression independent of relapse activity predictions using an interpretability algorithm. A total of 259 patients were evaluated, 58 (22%) of whom experienced at least one event of progression independent of relapse activity over a median follow-up of 4.2 years. The deep learning model demonstrated high performance (time-dependent concordance index = 0.72) with an accuracy of 78% in the original cohort and 72% in the external cohort for predicting the risk of progression independent of relapse activity. Incorporating the deep learning-derived cumulative probability of progression independent of relapses significantly improved an age-adjusted Cox regression model, raising Harrell's C index from 0.62 to 0.74. Interpretability revealed the frontoparietal cortex as a key region in predicting progression independent of relapse activity. In conclusion, our deep learning survival model, based on routine MRI at the first demyelinating attack, can accurately identify patients at high risk of progression independent of relapses and may serve as a valuable tool in clinical practice.

Keywords: brain MRI; deep learning; multiple sclerosis; predictive model; progression independent of relapse activity.

Graphical Abstract

Figure 1

Study design. DL model. (A) Overview of the proposed pipeline for the prediction of survival probabilities and their evaluation. The T2-FLAIR and T1-w 2D slices are used as input to the pre-trained EfficientNet-b0, whose extracted features are grouped in the discrete predictor to predict the discrete survivor function $\hat{S}(t),$ that is, the time-discrete interval-specific survival probabilities, i.e. the probabilities of not reaching PIRA. The predictor block consists of a fully connected layer, which outputs an 11-dimension vector, one for each time interval, followed by a sigmoid activation function, which outputs 11 (interval-specific) survival probabilities. These survival probabilities conform the survivor function, $\hat{S}(t),$ representing the probability of not reaching PIRA at each discrete time t (at each time interval). Then, the cumulative probability of having had the PIRA event by time t, i.e. the cumulative distribution function  $\hat{F}(t)$, is obtained as $1-\hat{S}(t).$ These interval-specific cumulative probabilities of reaching the event PIRA are then used to find the best threshold to identify, which patients will finally develop the event. Also, these interval-specific probabilities of PIRA are added as covariates in a Cox Proportional Hazards model predicting time to first PIRA (see the Material and methods section for more details). (B) EfficientNet-b0 architecture and the main layers forming the convolutional and residual blocks. Conforming the MBConv block there were depth-wise convolutional layers, which expand features instead of reducing them. After that, SE blocks were used to improve the quality of representations by explicitly modelling the inter-dependencies between the channels. Additionally, in between these two blocks a Swish activation was used, defined as f(x) = x · Sigmoid(x), that tends to work better than ReLU on deeper models. B, output feature maps; Conv, convolutional layer; DL, deep learning; FC, fully connected; GAP, global adaptive pooling; k, kernel size; M, input feature maps; MBConv, mobile inverted bottleneck; PIRA, progression independent of relapse activity; r, reduction ration of SE; s, stride; SE, squeeze-and-excitation; T1-w, T1-weighted image; T2-FLAIR, T2-weighted fluid-attenuated inversion recovery.

Figure 2

Patient flowchart. EDSS, expanded disability status scale; IC, informed consent; PIRA, progression independent of relapse activity.

Figure 3

DL-based predictions in the original cohort. (A) Overlay of Kaplan–Meier curve (grey) and DL-based survival predictions of PIRA (pink line) in the original cohort. The darker pink ribbon represents the 95% CI of the estimation, whereas the lighter pink one represents the 95% reference range, i.e. the DL-predicted PIRA survival probabilities in the population, from the 2.5th to the 97.5th percentiles. As observed, the DL-based predictions accurately mirrored the Kaplan–Meier curve, especially from symptom onset until the 8th year of follow-up. The mean time-dependent concordance index (c^td) across folds was 0.72 (range 0.68 to 0.78) and the mean IBS was 0.1 (SD 0.04). (B). DL-based predicted probabilities of PIRA at each time interval. Here, each dot in the graph represents the predicted probability of developing PIRA. Patients who developed PIRA during the follow-up had, on average, greater DL-predicted risk of developing PIRA at all time points: Mann–Whitney U-test (for comparison of medians): Interval 1: z (test statistic, large-sample normal [Z] approximation of the U statistic) = −4.501, P = 6.8·10⁻⁶; Interval 2: z = −4.465, P = 8·10⁻⁶; Interval 3: z = −4.168, P = 3.1·10⁻⁵; Interval 4: z = −3.940, P = 8.2·10⁻⁵; Interval 5: z = −3.977, P = 7·10⁻⁵; Interval 6: z = −3.896, P = 9.8·10⁻⁵; Interval 7: z = −3.860, P = 0.00011; Interval 8: z = −3.534, P = 0.00041; Interval 9: z = −3.373, P = 0.00075; Interval 10: z = −3.500, P = 0.00047; Interval 11: z = −3.400, P = 0.00064. Sample size for all comparisons: N = 259. DL, deep learning; IBS, integrated Brier score; SD, standard deviation; PIRA, progression independent of relapse activity.

Figure 4

Assessment of DL model performance and interpretability. (A) Comparison of ROC curves for the different interval-specific cumulative probability of a first PIRA event: the AUC for the first interval-specific probability is the greatest, although the difference with the rest of the curves did not reach statistical significance according to a χ² test, correcting for multiple comparisons (Šidák correction): χ² statistic (10 degrees of freedom) = 17.32, P = 0.0676. Sample size: N = 259. (B) Coefficient plot showing the independent contribution of age and lesion load at first attack plus the DL-based cumulative probabilities of PIRA, estimated at the first time interval, to predict PIRA through a Cox PH model (and Wald tests to assess individual covariate effects): age at first attack (in decades): aHR = 1.482 (1.041; 2.111), z (test statistic) = 2.18, P = 0.029; number of brain T2 lesions at first attack: aHR = 1.093 (1.014; 1.179), z = 2.31, P = 0.021; DL-based cumulative probabilities of PIRA: aHR = 1.449 (1.306; 1.607), z = 7.01, P < 0.001. Sample size for all analyses: N = 259. (C) Population-average maps of SHAP values, i.e. relevance maps (top row) and the most relevant areas for predicting PIRA (bottom row). Most relevant regions for PIRA prediction were the frontal and parietal cortex, lateral ventricles, periventricular, frontal and parietal WM. Cortical areas had a more negative relevance, lateral ventricles a more positive one, while a similar relevance value was observed in periventricular WM. Sample size for all analyses: N = 259. 95% CI, 95% confidence interval; aHR, adjusted hazard ratio; AUC, area under the ROC curve; Cox PH, Cox proportional hazards; DL, deep learning; PIRA, progression independent of relapse activity; ROC, receiver operating characteristic; WM, white matter.

Figure 5

DL-based predictions in the validation cohort. (A) Overlay of Kaplan–Meier curve (grey) and DL-based survival predictions of PIRA (orange line) in the validation cohort. The darker orange ribbon represents the 95% CI of the estimation, whereas the lighter orange one represents the 95% reference range, i.e. the DL-predicted PIRA survival probabilities in the population, from the 2.5th to the 97.5th percentiles. In the validation cohort, the DL-estimated survival probabilities of PIRA mirrored those of the Kaplan–Meier curve until the 5-year follow-up, when the two curves started to separate. (B) DL-based predicted probabilities of PIRA at each time interval. Here, each dot in the graph represents the predicted probability of developing PIRA. As in the original cohort, patients who developed PIRA during the follow-up had, on average, greater DL-predicted risk of developing PIRA at all time points. However, these differences did not reach statistical significance: Mann–Whitney U-test (for comparison of medians): Interval 1: z (test statistic, large-sample normal [Z] approximation of the U statistic) = −0.189, P = 0.87; Interval 2: z = −0.519, P = 0.63; Interval 3: z = −0.377, P = 0.73; Interval 4: z = −0.189, P = 0.87; Interval 5: z = −0.094, P = 0.95; Interval 6: z = −0.141, P = 0.91. Sample size for all comparisons: N = 32. DL, deep learning; PIRA, progression independent of relapse activity.