Quality Assurance using Outlier Detection on an Automatic Segmentation Method for the Cerebellar Peduncles

Abstract

Cerebellar peduncles (CPs) are white matter tracts connecting the cerebellum to other brain regions. Automatic segmentation methods of the CPs have been proposed for studying their structure and function. Usually the performance of these methods is evaluated by comparing segmentation results with manual delineations (ground truth). However, when a segmentation method is run on new data (for which no ground truth exists) it is highly desirable to efficiently detect and assess algorithm failures so that these cases can be excluded from scientific analysis. In this work, two outlier detection methods aimed to assess the performance of an automatic CP segmentation algorithm are presented. The first one is a univariate non-parametric method using a box-whisker plot. We first categorize automatic segmentation results of a dataset of diffusion tensor imaging (DTI) scans from 48 subjects as either a success or a failure. We then design three groups of features from the image data of nine categorized failures for failure detection. Results show that most of these features can efficiently detect the true failures. The second method-supervised classification-was employed on a larger DTI dataset of 249 manually categorized subjects. Four classifiers-linear discriminant analysis (LDA), logistic regression (LR), support vector machine (SVM), and random forest classification (RFC)-were trained using the designed features and evaluated using a leave-one-out cross validation. Results show that the LR performs worst among the four classifiers and the other three perform comparably, which demonstrates the feasibility of automatically detecting segmentation failures using classification methods.

Keywords: box-whisker plot; cerebellar peduncles; classification; outlier detection; quality assurance; segmentation.

Figure 1

One reference and nine segmentation failures in the first data set with 48 subjects: (a) is a successful segmentation of a subject as a reference. (b)–(j) are nine segmentation failures from nine subjects.

Figure 2

Volumes of six CPs of manual delineations of 10 subjects in the first dataset (red boxes) and automatic segmentations of the 48 subjects (blue boxes), respectively.

Figure 3

Means and standard deviations of FA and MD of the whole brains of the 48 subjects.

Figure 4

Means and standard deviations of the three Westin índices of the whole brains of the 48 subjects.

Figure 5

Brain mask features: volumes of the left, right, and whole brain masks (the left three boxplots) and the symmetry of the brain masks (the rightest boxplot) of the 48 subjects.