Objective Evaluation of Multiple Sclerosis Lesion Segmentation using a Data Management and Processing Infrastructure

Abstract

We present a study of multiple sclerosis segmentation algorithms conducted at the international MICCAI 2016 challenge. This challenge was operated using a new open-science computing infrastructure. This allowed for the automatic and independent evaluation of a large range of algorithms in a fair and completely automatic manner. This computing infrastructure was used to evaluate thirteen methods of MS lesions segmentation, exploring a broad range of state-of-theart algorithms, against a high-quality database of 53 MS cases coming from four centers following a common definition of the acquisition protocol. Each case was annotated manually by an unprecedented number of seven different experts. Results of the challenge highlighted that automatic algorithms, including the recent machine learning methods (random forests, deep learning, …), are still trailing human expertise on both detection and delineation criteria. In addition, we demonstrate that computing a statistically robust consensus of the algorithms performs closer to human expertise on one score (segmentation) although still trailing on detection scores.

Figure 1

Illustration of an MS patient delineations overlaid on the 3D FLAIR image. (a–g) Individual manual delineations of MS lesions from each of the experts, (h) consensus segmentation considered as the ground truth.

Figure 2

Graphical results illustration of automatic clustering of average results for each team and expert into three groups (scatter plots of pairs of two evaluation parameters: (a) Dice and F₁ scores, (b) surface distance and F₁ scores, (c) surface distance and Dice scores). Legend: blue crosses: group 1 (always containing only the seven experts even though the clustering is automatic), green crosses: group 2 (best performing algorithms), red crosses: group 3 (lower “quality” algorithms). Team numbers associated with each point on the graph are indicated as labels. Team fusion indicates a composite segmentation result further discussed in Section 2.5.

Figure 3

Dice scores (a) F₁ scores (b) and average surface distances (c) with respect to the consensus per team for each center and averaged over all centers.

Figure 4

Dice scores (a) F₁ scores (b) and average surface distances (c) with respect to the consensus for each center and averaged over all centers for composite Team fusion with respect to the average experts agreement level.

Figure 5

Link between average scores of all methods and number of lesions (first column) and total lesion load (cm³, second column). First line: Dice score, second line: F₁ score, third line: average surface distance.

Figure 6

Individual lesion detection rate (average over all methods) as a function of lesion size. X-axis: individual lesion volume on a logarithmic scale. Y-axis: detection rate (in percentages, number of teams detecting a lesion of this volume over all patients).

Figure 7

FLI-IAM architecture.

Figure 8

Workflow for database and computing platform integration.

Figure 9

Illustration of overlap-based segmentation evaluation: quantities used for measures computation. A denotes the evaluated segmentation, G the ground truth, and B the image domain.

Algorithm 1

TP_G computation algorithm.