Simultaneous lesion and brain segmentation in multiple sclerosis using deep neural networks

Abstract

Segmentation of white matter lesions and deep grey matter structures is an important task in the quantification of magnetic resonance imaging in multiple sclerosis. In this paper we explore segmentation solutions based on convolutional neural networks (CNNs) for providing fast, reliable segmentations of lesions and grey-matter structures in multi-modal MR imaging, and the performance of these methods when applied to out-of-centre data. We trained two state-of-the-art fully convolutional CNN architectures on the 2016 MSSEG training dataset, which was annotated by seven independent human raters: a reference implementation of a 3D Unet, and a more recently proposed 3D-to-2D architecture (DeepSCAN). We then retrained those methods on a larger dataset from a single centre, with and without labels for other brain structures. We quantified changes in performance owing to dataset shift, and changes in performance by adding the additional brain-structure labels. We also compared performance with freely available reference methods. Both fully-convolutional CNN methods substantially outperform other approaches in the literature when trained and evaluated in cross-validation on the MSSEG dataset, showing agreement with human raters in the range of human inter-rater variability. Both architectures showed drops in performance when trained on single-centre data and tested on the MSSEG dataset. When trained with the addition of weak anatomical labels derived from Freesurfer, the performance of the 3D Unet degraded, while the performance of the DeepSCAN net improved. Overall, the DeepSCAN network predicting both lesion and anatomical labels was the best-performing network examined.

Figure 1

The DeepSCAN architecture used in this paper for lesion and brain-structure segmentation.

Figure 2

Example segmentations of a case from the Insel32 dataset, axial view. (A) FLAIR image, plus segmentations from (B) manual raters, (C) LPA, (D) LGA, (E) nicMSlesions and (F) nnUnet (all labels) (G) DeepSCAN (just lesions) (H) nnUnet (just lesions) (I) DeepSCAN (all labels). Segmentations from LGA, LPA, and nnUnet (all labels) show erroneous segmentation of healthy-appearing tissue near the ventricles (cyan arrows). nicMSlesions labels healthy cortical grey matter as lesion tissue (cyan arrows) DeepSCAN (all labels) identified two subtle lesions (violet arrows) missed by the manual raters but subsequently confirmed by our experienced neuroradiologists.

Figure 3

Boxplots of Dice coefficients for selected grey matter structures, for classifiers trained on the Insel90 dataset, and validated on the Insel32 (in sample) and MSSEG (out of sample) datasets.

Figure 4

An axial slice of case 1016SACH from the MSSEG dataset. Top left, FLAIR, top right, fused ’ground truth’, bottom left, DeepSCAN segmentation, trained on the Insel90 dataset, bottom right, nnUnet segmentation, trained on the same Insel90 dataset. nnUnet model incorrectly labels parts of Caudate nucleus as Thalamus, and incorrectly places boundary between Pallidum and Putamen.