Reduced accuracy of MRI deep grey matter segmentation in multiple sclerosis: an evaluation of four automated methods against manual reference segmentations in a multi-center cohort

Alexandra de Sitter¹, Tom Verhoeven², Jessica Burggraaff³, Yaou Liu², Jorge Simoes², Serena Ruggieri^{4

5}, Miklos Palotai⁶, Iman Brouwer², Adriaan Versteeg², Viktor Wottschel², Stefan Ropele⁷, Mara A Rocca^{8

9}, Claudio Gasperini⁵, Antonio Gallo¹⁰, Marios C Yiannakas¹¹, Alex Rovira¹², Christian Enzinger¹³, Massimo Filippi^{8

9

14

15}, Nicola De Stefano¹⁶, Ludwig Kappos¹⁷, Jette L Frederiksen¹⁸, Bernard M J Uitdehaag³, Frederik Barkhof^{2

19}, Charles R G Guttmann⁶, Hugo Vrenken²; MAGNIMS Study Group

Affiliations

¹ Department of Radiology and Nuclear Medicine, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands. A.deSitter@amsterdamumc.nl.
² Department of Radiology and Nuclear Medicine, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands.
³ Department of Neurology, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, Amsterdam, The Netherlands.
⁴ Department of Human Neurosciences, "Sapienza" University of Rome, Rome, Italy.
⁵ Department of Neurosciences, San Camillo Forlanini Hospital, Rome, Italy.
⁶ Center for Neurological Imaging, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
⁷ Department of Neurology, Medical University of Graz, Graz, Austria.
⁸ Neuroimaging Research Unit, Division of Neuroscience, Institute of Experimental Neurology, Milan, Italy.
⁹ Neurology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy.
¹⁰ Division of Neurology and MRI Research Center, Department of Medical, Surgical, Neurologic, Metabolic and Aging Sciences, University of Campania "Luigi Vanvitelli", Naples, Italy.
¹¹ Research Unit, Queen Square MS Centre, Department of Neuroinflammation, UCL Queen Square Institute of Neurology, University College London, London, UK.
¹² Unitat de Ressonància Magnètica (Servei de Radiologia), Hospital Universitari Vall D'Hebron, Autonomous University of Barcelona, Barcelona, Spain.
¹³ Division of Neuroradiology, Vascular and Interventional Radiology, Department of Radiology Medical, University of Graz, Graz, Austria.
¹⁴ Neurophysiology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy.
¹⁵ Vita-Salute San Raffaele University, Milan, Italy.
¹⁶ Department of Neurological and Behavioural Sciences, University of Siena, Siena, Italy.
¹⁷ Department of Neurology, University Hospital, Kantonsspital, Basel, Switzerland.
¹⁸ Department of Neurology, Glostrup University Hospital Copenhagen, Copenhagen, Denmark.
¹⁹ Institutes of Neurology and Healthcare Engineering, UCL London, London, UK.

PMID: 32621103
PMCID: PMC7674567
DOI: 10.1007/s00415-020-10023-1

Reduced accuracy of MRI deep grey matter segmentation in multiple sclerosis: an evaluation of four automated methods against manual reference segmentations in a multi-center cohort

Alexandra de Sitter et al. J Neurol. 2020 Dec.

. 2020 Dec;267(12):3541-3554.

doi: 10.1007/s00415-020-10023-1. Epub 2020 Jul 3.

Authors

Affiliations

¹ Department of Radiology and Nuclear Medicine, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands. A.deSitter@amsterdamumc.nl.
² Department of Radiology and Nuclear Medicine, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, De Boelelaan 1117, 1081 HV, Amsterdam, The Netherlands.
³ Department of Neurology, MS Center Amsterdam, Amsterdam Neuroscience, Amsterdam UMC, Location VUmc, Amsterdam, The Netherlands.
⁴ Department of Human Neurosciences, "Sapienza" University of Rome, Rome, Italy.
⁵ Department of Neurosciences, San Camillo Forlanini Hospital, Rome, Italy.
⁶ Center for Neurological Imaging, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
⁷ Department of Neurology, Medical University of Graz, Graz, Austria.
⁸ Neuroimaging Research Unit, Division of Neuroscience, Institute of Experimental Neurology, Milan, Italy.
⁹ Neurology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy.
¹⁰ Division of Neurology and MRI Research Center, Department of Medical, Surgical, Neurologic, Metabolic and Aging Sciences, University of Campania "Luigi Vanvitelli", Naples, Italy.
¹¹ Research Unit, Queen Square MS Centre, Department of Neuroinflammation, UCL Queen Square Institute of Neurology, University College London, London, UK.
¹² Unitat de Ressonància Magnètica (Servei de Radiologia), Hospital Universitari Vall D'Hebron, Autonomous University of Barcelona, Barcelona, Spain.
¹³ Division of Neuroradiology, Vascular and Interventional Radiology, Department of Radiology Medical, University of Graz, Graz, Austria.
¹⁴ Neurophysiology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy.
¹⁵ Vita-Salute San Raffaele University, Milan, Italy.
¹⁶ Department of Neurological and Behavioural Sciences, University of Siena, Siena, Italy.
¹⁷ Department of Neurology, University Hospital, Kantonsspital, Basel, Switzerland.
¹⁸ Department of Neurology, Glostrup University Hospital Copenhagen, Copenhagen, Denmark.
¹⁹ Institutes of Neurology and Healthcare Engineering, UCL London, London, UK.

PMID: 32621103
PMCID: PMC7674567
DOI: 10.1007/s00415-020-10023-1

Abstract

Background: Deep grey matter (DGM) atrophy in multiple sclerosis (MS) and its relation to cognitive and clinical decline requires accurate measurements. MS pathology may deteriorate the performance of automated segmentation methods. Accuracy of DGM segmentation methods is compared between MS and controls, and the relation of performance with lesions and atrophy is studied.

Methods: On images of 21 MS subjects and 11 controls, three raters manually outlined caudate nucleus, putamen and thalamus; outlines were combined by majority voting. FSL-FIRST, FreeSurfer, Geodesic Information Flow and volBrain were evaluated. Performance was evaluated volumetrically (intra-class correlation coefficient (ICC)) and spatially (Dice similarity coefficient (DSC)). Spearman's correlations of DSC with global and local lesion volume, structure of interest volume (ROIV), and normalized brain volume (NBV) were assessed.

Results: ICC with manual volumes was mostly good and spatial agreement was high. MS exhibited significantly lower DSC than controls for thalamus and putamen. For some combinations of structure and method, DSC correlated negatively with lesion volume or positively with NBV or ROIV. Lesion-filling did not substantially change segmentations.

Conclusions: Automated methods have impaired performance in patients. Performance generally deteriorated with higher lesion volume and lower NBV and ROIV, suggesting that these may contribute to the impaired performance.

Keywords: Atrophy; Automated segmentation methods; Deep grey matter; Multiple sclerosis.

PubMed Disclaimer

Conflict of interest statement

AdS is employed on a project sponsored by a research grant from Teva Pharmaceuticals (grant to H. Vrenken and F. Barkhof). TV has no disclosures. YL has no disclosures related to this paper. JB has no disclosures. JS has no disclosures. SR received fees as invited speaker or travel expenses for attending meeting from Biogen, Merck-Serono, Teva, Sanofi, Novartis. MP has no disclosures. IB is partly employed on projects sponsored by research grants from Teva Pharmaceuticals and Novartis Pharma (grants to HV and FB). AV has no disclosures. VW has no disclosures related to this paper. SR has no disclosures related to this paper. MAR received speakers’ honoraria from Biogen Idec, Novartis, Genzyme, Sanofi-Aventis, Teva, Merck Serono, Roche and Celgene and receives research support from the Italian Ministry of Health and Fondazione Italiana Sclerosi Multipla. CG received fees as speaker for Bayer-Schering Pharma, Sanofi-Aventis, Genzyme, Biogen, Teva, Novartis, and Merck-Serono, and received a grant for research by Teva. AG has no disclosures related to this paper. MCY has no disclosures. AR has no disclosures related to this paper. CE has no disclosures related to this paper. MF is Editor-in-Chief of the Journal of Neurology; received compensation for consulting services and/or speaking activities from Biogen Idec, Merck-Serono, Novartis, Teva Pharmaceutical Industries; and receives research support from Biogen Idec, Merck-Serono, Novartis, Teva Pharmaceutical Industries, Roche, Italian Ministry of Health, Fondazione Italiana Sclerosi Multipla, and ARiSLA (Fondazione Italiana di Ricerca per la SLA). NDS has no disclosures related to this paper. LK has no disclosures related to this paper. JLF declares personal fees from Biogen Idec, Merck Serono and Sanofi-Aventis, participation in scientific advisory boards for Almiral, Genzyme and Novartis, personal fees (speaker honoraria) from Biogen, Merck, Santhera and Teva, all unrelated to this paper. FB has received compensation for consulting services and/or speaking activities from Bayer, Biogen Idec, Merck Serono, Novartis, Roche, Teva, Bracco and IXICO. CRGG has received support from the National Multiple Sclerosis Society, the International Progressive Multiple Sclerosis Alliance, the U.S. Office for Naval Research, Mobilengine (free use of platform and programming by Mobilengine Engineers), NIH, as well as travel support from Roche Pharmaceuticals; C.R.G owns stock in Roche, Novartis, GSK, Alnylam, Protalix Biotherapeutics, Arrowhead Pharmaceuticals, Cocrystal Pharma, Sangamo Therapeutics. HV has received research grants from Pfizer, MerckSerono, Novartis and Teva, speaker honoraria from Novartis, and consulting fees from MerckSerono; all funds were paid directly to his Institution.

Figures

**Fig. 1**
Method for lesion load calculation within a set border. a a distance field is created around DGM structure, in this case the caudate nucleus. b Distance is set around the DGM structure, seen in grey. c An overlay is created between the DGM border and the lesion mask in T1 space. In grey the lesion border is shown, in red the lesions without overlap with the border and in yellow the lesions with an overlap in the border

**Fig. 2**
T1 weighted images and segmentation of majority voting, FSL-FIRST, Freesurfer, GIF and volBrain. Segmentations of both left and right hemisphere and for all three structures; caudate, putamen and thalamus

**Fig. 3**
Majority voting segmentation volume and volume by automatic segmentation are given for each deep gray matter structure and segmentation method. Volumes are given in milliliters

**Fig. 4**
Dice similarity coefficients between segmentations from majority voting and each automated method per DGM structure for both healthy controls (HC) and patients (green)

**Fig. 5**
Dice similarity coefficients versus lesion load, represented per DGM structure and segmentation method and left (blue) and right (green) hemisphere

See this image and copyright information in PMC

References

1. Derakhshan M, Caramanos Z, Giacomini PS, Narayanan S, Maranzano J, Francis SJ, et al. Evaluation of automated techniques for the quantification of grey matter atrophy in patients with multiple sclerosis. Neuroimage. 2010;52(4):1261–1267. doi: 10.1016/j.neuroimage.2010.05.029. - DOI - PubMed
1. Eshaghi A, Marinescu RV, Young AL, Firth NC, Prados F, Jorge Cardoso M, et al. Progression of regional grey matter atrophy in multiple sclerosis. Brain. 2018;141(6):1665–1677. doi: 10.1093/brain/awy088. - DOI - PMC - PubMed
1. Modica CM, Bergsland N, Dwyer MG, Ramasamy DP, Carl E, Zivadinov R, et al. Cognitive reserve moderates the impact of subcortical gray matter atrophy on neuropsychological status in multiple sclerosis. Mult Scler. 2016;22(1):36–42. doi: 10.1177/1352458515579443. - DOI - PubMed
1. Schoonheim MM, Ciccarelli O. The value of including thalamic atrophy as a clinical trial endpoint in multiple sclerosis. Neurology. 2018;90(15):677–678. doi: 10.1212/WNL.0000000000005279. - DOI - PubMed
1. Amiri H, de Sitter A, Bendfeldt K, Battaglini M, Gandini Wheeler-Kingshott CAM, Calabrese M, et al. Urgent challenges in quantification and interpretation of brain grey matter atrophy in individual MS patients using MRI. Neuroimage Clin. 2018;19:466–475. doi: 10.1016/j.nicl.2018.04.023. - DOI - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
Medical
- MedlinePlus Health Information

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reduced accuracy of MRI deep grey matter segmentation in multiple sclerosis: an evaluation of four automated methods against manual reference segmentations in a multi-center cohort

Affiliations

Reduced accuracy of MRI deep grey matter segmentation in multiple sclerosis: an evaluation of four automated methods against manual reference segmentations in a multi-center cohort

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

MeSH terms

LinkOut - more resources

Full Text Sources

Medical