A comparison of automated segmentation and manual tracing for quantifying hippocampal and amygdala volumes

Abstract

Large databases of high-resolution structural MR images are being assembled to quantitatively examine the relationships between brain anatomy, disease progression, treatment regimens, and genetic influences upon brain structure. Quantifying brain structures in such large databases cannot be practically accomplished by expert neuroanatomists using hand-tracing. Rather, this research will depend upon automated methods that reliably and accurately segment and quantify dozens of brain regions. At present, there is little guidance available to help clinical research groups in choosing such tools. Thus, our goal was to compare the performance of two popular and fully automated tools, FSL/FIRST and FreeSurfer, to expert hand tracing in the measurement of the hippocampus and amygdala. Volumes derived from each automated measurement were compared to hand tracing for percent volume overlap, percent volume difference, across-sample correlation, and 3-D group-level shape analysis. In addition, sample size estimates for conducting between-group studies were computed for a range of effect sizes. Compared to hand tracing, hippocampal measurements with FreeSurfer exhibited greater volume overlap, smaller volume difference, and higher correlation than FIRST, and sample size estimates with FreeSurfer were closer to hand tracing. Amygdala measurement with FreeSurfer was also more highly correlated to hand tracing than FIRST, but exhibited a greater volume difference than FIRST. Both techniques had comparable volume overlap and similar sample size estimates. Compared to hand tracing, a 3-D shape analysis of the hippocampus showed FreeSurfer was more accurate than FIRST, particularly in the head and tail. However, FIRST more accurately represented the amygdala shape than FreeSurfer, which inflated its anterior and posterior surfaces.

Fig. 1

A selection of manually segmented hippocampal slices of a representative subject.

Fig. 2

A selection of manually segmented amygdala slices of a representative subject.

Fig. 3

Method for transforming brains from native space to standard space to perform segmentation with FreeSurfer and FIRST and then transform back to native space for volume and shape comparisons. The function tkregister2, available in the FreeSurfer library, performs transformation with minimal interpolation which typically leads to distortion. Therefore regions retain their original dimensions.

Fig. 4

Percent volume overlap (Dice's coefficient) between FreeSurfer segmentation and manual tracing is greater than the overlap between FIRST and manual tracing for the left and right hippocampus. In the amygdala, the overlap is not different between manual tracing and either of the two automated methods. Percent volume overlap is greater in the hippocampus than the amygdala regardless of the segmentation method.

Fig. 5

In the left and right hippocampus, the percent volume difference relative to manual tracing, is smaller with FreeSurfer than FIRST. In the left and right amygdala, the volume difference with manual tracing, is smaller with FIRST than FreeSurfer. There was a smaller FreeSurfer-Manual volume difference in the L-amygdala than the R-amygdala. No other laterality differences were detected.

Fig. 6

(A) Hippocampal volume derived from FreeSurfer segmentation is highly correlated with manual tracing (R=0.82; p<10⁻⁹). (B) Hippocampal volume derived from FSL/FIRST segmentation is correlated with manual tracing (R=0.66; p<10⁻⁵).

Fig. 7

(A) Amygdala volume derived from FreeSurfer is correlated with manual tracing (R=0.56; p<0.0005). (B) Amygdala volume derived from FSL/FIRST was poorly correlated with manual tracing (R=0.24; p>0.13).

Fig. 8

Bland–Altman mean difference plots for hippocampal and amygdala volumes.

Fig. 9

Shape analysis of the hippocampus in 3-D where difference maps show the distance between segmentation contours thresholded from 0.2mmto 2.0 mm. Vectors indicate the directionality between the corresponding mesh vertices of the two segmentation methods. Ellipsoids indicate the [x, y, z] components of variance introduced by the automated segmentation. Significance maps (permutation corrected, p<0.05) highlight shape differences between automated segmentation and manual tracing. (A) FIRST compared to manual tracing, and (B) FreeSurfer compared to manual tracing.

Fig. 10

Shape analysis of the amygdala in 3-D where difference maps show the distance between segmentation contours thresholded from 0.2 mm to 2.0 mm. Vectors indicate the directionality between the corresponding mesh vertices of the two segmentation methods. Ellipsoids indicate the [x, y, z] components of variance introduced by the automated segmentation. Significance maps (permutation corrected, p<0.05) highlight shape differences between automated segmentation and manual tracing. (A) FIRST compared to manual tracing, and (B) FreeSurfer compared to manual tracing.

Fig. 11

FreeSurfer segmentation had the power to detect differences in hippocampal volume between groups for a range of effect sizes (power=0.8; alpha=0.05) with negligible increase in sample size relative to manual tracing. On the other hand, FIRST segmentation required a modest increase in sample size particularly to detect relatively small effects.

Fig. 12

FreeSurfer and FIRST segmentation had roughly equal power in detecting differences in amygdala volume between groups for a range of effect sizes (power=0.8; alpha=0.05 but, both methods required a modest increase in sample size relative to manual tracing, particularly to detect relatively small effects. Note that sample size estimates were derived solely from standard deviation with correlation having no role in the estimation process.

Fig. 13

FreeSurfer showed a 9% reduction in hippocampal volume for participants diagnosed with major depressive disorder (MDD) compared to a matched Control Group. However, FIRST did not show differences between MDD and Control groups.