Deciphering multiple sclerosis disability with deep learning attention maps on clinical MRI

Abstract

The application of convolutional neural networks (CNNs) to MRI data has emerged as a promising approach to achieving unprecedented levels of accuracy when predicting the course of neurological conditions, including multiple sclerosis, by means of extracting image features not detectable through conventional methods. Additionally, the study of CNN-derived attention maps, which indicate the most relevant anatomical features for CNN-based decisions, has the potential to uncover key disease mechanisms leading to disability accumulation. From a cohort of patients prospectively followed up after a first demyelinating attack, we selected those with T1-weighted and T2-FLAIR brain MRI sequences available for image analysis and a clinical assessment performed within the following six months (N = 319). Patients were divided into two groups according to expanded disability status scale (EDSS) score: ≥3.0 and < 3.0. A 3D-CNN model predicted the class using whole-brain MRI scans as input. A comparison with a logistic regression (LR) model using volumetric measurements as explanatory variables and a validation of the CNN model on an independent dataset with similar characteristics (N = 440) were also performed. The layer-wise relevance propagation method was used to obtain individual attention maps. The CNN model achieved a mean accuracy of 79% and proved to be superior to the equivalent LR-model (77%). Additionally, the model was successfully validated in the independent external cohort without any re-training (accuracy = 71%). Attention-map analyses revealed the predominant role of frontotemporal cortex and cerebellum for CNN decisions, suggesting that the mechanisms leading to disability accrual exceed the mere presence of brain lesions or atrophy and probably involve how damage is distributed in the central nervous system.

Keywords: Attention maps; Deep learning; Disability; Multiple sclerosis; Structural MRI.

Fig. 1

Pipeline followed in this work. a For training and inference, whole brain input is evaluated by the classifier model to predict the probability belonging to each disability status. We set a threshold of 0.5 for this probability. After inference, the probability obtained from the model is fed into the LRP algorithm to backpropagate through the CNN and obtain the attention map. As shown in b, the training procedure is only performed with the in-house database in a cross-validation strategy, hence obtaining different models. Afterwards, these models are used to evaluate the external database, where the final classification decision is obtained with the majority voting of the different models. P: probability, f: fold, LRP: layer-wise relevance propagation.

Figure 2

Network architecture. Residual convolutional neural network architecture. Res block: residual block, conv: convolutional layer, k: kernels, GAP: global adaptive max pooling, ReLU: rectified linear unit.

Fig. 3

Example of an individual attention map analysis. This MS patient was correctly classified as moderate disability with an EDSS = 6.0. a Different T2-FLAIR axial slices with their corresponding computed attention map. b Relevant-voxel accumulation by anatomical area, obtained from the product of the binarised attention map by the brain anatomical parcellation. In this case, the frontal cortex and periventricular WM were the most relevant areas leading the decision.

Fig. 4

Class-average attention map analyses. a Class-average attention maps (TP, FP, TN, FN) binarised at 95% percentile. b Brain parcellation with the mean attention value (normalised across groups) attributed to each anatomical region. c Mean voxel accumulation by each anatomical area depending on the class-average group.

Fig. 5

Voxel-wise regression analyses. Average map for the TP, FP, TN and FN of the T1-w scan, lesion probability map, Jacobian determinant, LRP heatmap and the R-squared (R²) map obtained from the voxel-wise regression model built with the individual attention maps as dependent variable, and the individual lesion masks and individual Jacobian determinants as explanatory ones. The partial R² on each separate variable is also represented. The Jacobian determinant is represented in terms of expansion (values > 1.0) and compression (values < 1.0).