Task-based evaluation of segmentation algorithms for diffusion-weighted MRI without using a gold standard

Abstract

In many studies, the estimation of the apparent diffusion coefficient (ADC) of lesions in visceral organs in diffusion-weighted (DW) magnetic resonance images requires an accurate lesion-segmentation algorithm. To evaluate these lesion-segmentation algorithms, region-overlap measures are used currently. However, the end task from the DW images is accurate ADC estimation, and the region-overlap measures do not evaluate the segmentation algorithms on this task. Moreover, these measures rely on the existence of gold-standard segmentation of the lesion, which is typically unavailable. In this paper, we study the problem of task-based evaluation of segmentation algorithms in DW imaging in the absence of a gold standard. We first show that using manual segmentations instead of gold-standard segmentations for this task-based evaluation is unreliable. We then propose a method to compare the segmentation algorithms that does not require gold-standard or manual segmentation results. The no-gold-standard method estimates the bias and the variance of the error between the true ADC values and the ADC values estimated using the automated segmentation algorithm. The method can be used to rank the segmentation algorithms on the basis of both the ensemble mean square error and precision. We also propose consistency checks for this evaluation technique.

Figure 1

The regression lines estimated using the no-gold-standard method for the three segmentation algorithms and for the two simulated lesion datasets: DS1 and DS2. In these two datasets, the true ADC values were sampled from a normal distribution. The solid line is generated using the estimated linear model parameters, and the dashed line denotes the estimated standard deviation. The scatter plots show the computed ADC values vs. the true ADC value. ADC values are in units of mm²/10³s. Note that although we have plotted the true ADC value on the x axis of the graph, this information was not used in computing the linear model parameters.

Figure 2

Consistency check: The Q-Q plot comparing the histogram of automated ADC values to the theoretically-estimated distribution for the clustering, MLE and EM-based segmentation algorithms and for the two simulated lesion datasets: DS1 and DS2. The quantiles of the theoretically-estimated distribution and the measured ADC distribution are plotted along the x and y axis, respectively. The 45° line is also plotted for visual convenience. We observe that the Q-Q plot lies approximately along the 45° line in all the scenarios, thus confirming that the measured and estimated distributions are consistent with each other.

Figure 3

The regression lines estimated using the no-gold-standard method for the three segmentation algorithms and for the two simulated lesion datasets: DS3 and DS4. The solid line is generated using the estimated linear model parameters, and the dashed line denotes the estimated standard deviation. The scatter plots show the computed ADC values vs. the true ADC value. ADC values are in units of mm²/10³s.

Figure 4

Consistency check: The Q-Q plot comparing the histogram of automated ADC values to the theoretically-estimated distribution for the clustering, MLE and EM-based segmentation algorithms and for the two simulated lesion datasets: DS3 and DS4.