Longitudinal multiple sclerosis lesion segmentation: Resource and challenge

Abstract

In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website² as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

Keywords: Magnetic resonance imaging; Multiple sclerosis.

Figure 1

Shown are the preprocessed (a) MPRAGE, (b) FLAIR, (c) T₂-w, and (d) PD-w images for a single time-point from one of the provided Training Set subjects. The corresponding manual delineations by our two raters are shown in (e) for Rater #1 and (f) for Rater #2.

Figure 2

Delineations are shown for a sample slice from the preprocessed MPRAGE, FLAIR, and T₂-w images for a time-point of a test data set, followed by our Consensus Delineation and the results for the top eight delineations as ranked by their Dice Score with the Consensus. For ease of reference, a grid has been added underneath the delineations. The bottom eight delineations, as ranked by their Dice Score with the Consensus, can be see in Fig. 3.

Figure 3

Delineations are shown for a sample slice from the preprocessed MPRAGE, FLAIR, and T₂-w images for a time-point of a test data set, followed by our Consensus Delineation and the results for the bottom eight delineations as ranked by their Dice Score with the Consensus. For ease of reference, a grid has been added underneath the delineations. The top eight delineations, as ranked by their Dice Score with the Consensus, can be see in Fig. 2.

Figure 4

The plot shows a least squares linear regression fit between the lesion load estimated by each of the Segmentations and that from the Consensus Delineation. The dashed line represents a line of unit slope. All volumes are in mm³.

Figure 5

Each subplot shows the range of values for the true positive rate (TPR) and the positive predictive value (PPV) between eight of the Segmentations and the Consensus Delineation. The top plot shows the top eight Segmentations as ranked by the Dice overlap, and the bottom plot shows the remaining eight Segmentations. The desirable point on each of the subplots is the upper right hand corner, where TPR is 1 and PPV is 1.

Figure 6

Each subplot shows the range of values for the lesion true positive rate (LTPR) and the lesion false positive rate (LFPR) between eight of the Segmentations and the Consensus Delineation. The top plot shows the top eight Segmentations as ranked by the Dice overlap, and the bottom plot shows the remaining eight Segmentations. The desirable point on each of the subplots is the lower right hand corner, where LTPR is 1 and LFPR is 0.