Comparative performance evaluation of automated segmentation methods of hippocampus from magnetic resonance images of temporal lobe epilepsy patients

Abstract

Purpose: Segmentation of the hippocampus from magnetic resonance (MR) images is a key task in the evaluation of mesial temporal lobe epilepsy (mTLE) patients. Several automated algorithms have been proposed although manual segmentation remains the benchmark. Choosing a reliable algorithm is problematic since structural definition pertaining to multiple edges, missing and fuzzy boundaries, and shape changes varies among mTLE subjects. Lack of statistical references and guidance for quantifying the reliability and reproducibility of automated techniques has further detracted from automated approaches. The purpose of this study was to develop a systematic and statistical approach using a large dataset for the evaluation of automated methods and establish a method that would achieve results better approximating those attained by manual tracing in the epileptogenic hippocampus.

Methods: A template database of 195 (81 males, 114 females; age range 32-67 yr, mean 49.16 yr) MR images of mTLE patients was used in this study. Hippocampal segmentation was accomplished manually and by two well-known tools (FreeSurfer and hammer) and two previously published methods developed at their institution [Automatic brain structure segmentation (ABSS) and LocalInfo]. To establish which method was better performing for mTLE cases, several voxel-based, distance-based, and volume-based performance metrics were considered. Statistical validations of the results using automated techniques were compared with the results of benchmark manual segmentation. Extracted metrics were analyzed to find the method that provided a more similar result relative to the benchmark.

Results: Among the four automated methods, ABSS generated the most accurate results. For this method, the Dice coefficient was 5.13%, 14.10%, and 16.67% higher, Hausdorff was 22.65%, 86.73%, and 69.58% lower, precision was 4.94%, -4.94%, and 12.35% higher, and the root mean square (RMS) was 19.05%, 61.90%, and 65.08% lower than LocalInfo, FreeSurfer, and hammer, respectively. The Bland-Altman similarity analysis revealed a low bias for the ABSS and LocalInfo techniques compared to the others.

Conclusions: The ABSS method for automated hippocampal segmentation outperformed other methods, best approximating what could be achieved by manual tracing. This study also shows that four categories of input data can cause automated segmentation methods to fail. They include incomplete studies, artifact, low signal-to-noise ratio, and inhomogeneity. Different scanner platforms and pulse sequences were considered as means by which to improve reliability of the automated methods. Other modifications were specially devised to enhance a particular method assessed in this study.

FIG. 1.

The automatic techniques are categorized into four groups: (1) mapping and registration based on single or multiatlas, (2) pattern-matching based on information registration, (3) shape-fitting based on energy minimization, and (4) labeling assignment based on pattern recognition and machine learning. A fully automatic technique is chosen for each group. By statistical analysis, the accuracy, reliability, and reproducibility are assessed using a large database of mTLE MRI studies. The best performing method was selected and the input conditions of those cases with unacceptable segmentation outcomes were evaluated. Some modifications were specially devised to enhance a particular method assessed in this study.

FIG. 2.

Flowchart of the work: (1) skull stripping, (2) hippocampal segmentation by manual and automatic segmentation methods, (3) statistical validation by voxel, distance, and volume measures, and (4) analysis of the results based on the input data.

FIG. 3.

Skull-stripping steps: (A) input images, (B) brain contouring, and (C) removal of nonbrain tissues.

FIG. 4.

Manual segmentation of the hippocampi using coronal T1-weighted MR images and a previously established segmentation protocol.

FIG. 5.

Surface rendering of the cortex (left) and segmented hippocampi (right) of a 52 years old female with mTLE. The hippocampal segmentation is performed using the ABSS method.

FIG. 6.

Volume waveforms vs subjects for the right and left hippocampi (patients with Dice > 4.0), for the automated and manual methods.

FIG. 7.

A comparison of manual and automatic hippocampal segmentation methods. The figure shows an intermediate (first row, first column) section of a skull-stripped T1 image of a mTLE patient. The first overlay (first row, second column) shows the manual segmentation, the second overlay (first row, third column) shows the ABSS segmentation, the third overlay (second row, first column) shows the LocalInfo segmentation, the fourth overlay (second row, second column) shows the FreeSurfer segmentation, and the last overlay (second row, third column) shows the hammer segmentation.

FIG. 8.

Hippocampal volumes (mm³) estimated by the manual and automated methods for the left and right hippocampi and their linear regression.

FIG. 9.

Bland–Altman plots comparing manual results versus the following: ABSS (first row, first column), LocalInfo (first row, second column), FreeSurfer (second row, first column), and hammer (second row, second column) results for the left and right hippocampi.

FIG. 10.

Dice coefficient of the four automatic segmentation methods (FreeSurfer, hammer, LocalInfo, ABSS) vs case number for the following: (A) all 195 mTLE patients and (B) 157 mTLE patients with Dice > 0.4. Since those cases with unacceptable automated segmentation outcomes were evaluated separately, they were removed in (B) of this figure and the subsequent figures.

FIG. 11.

Precision (A), Hausdorff distance (B), and RMS (C) of the four automatic segmentation methods (ABSS, LocalInfo, Freesurfer, hammer) vs case number for 157 mTLE patients with Dice >0.4, sorted by case numbers.

FIG. 12.

Graphical depiction of groups using Dice (first row, first column), Hausdorff (first row, second column), precision (second row, first column), and RMS (second row, second column) on 157 subjects by quartile. Outliers are plotted by a plus sign.

FIG. 13.

Visual comparison of the hippocampus volumes (mm³) estimated by the automated segmentation methods vs ground truth (manual segmentation).

FIG. 14.

Segmentation results of other methods in cases that they outperformed ABSS. For subject numbers 107, 166, and 23, hammer, FreeSurfer, and LocalInfo, respectively, produced better segmentation than ABSS.

FIG. 15.

Four categories of input images in which segmentation output is unfavorable: (A) incomplete image set, (B) image with artifact, (C) noisy, low-quality image, and (D) inhomogeneous image.