A reproducible evaluation of ANTs similarity metric performance in brain image registration

Abstract

The United States National Institutes of Health (NIH) commit significant support to open-source data and software resources in order to foment reproducibility in the biomedical imaging sciences. Here, we report and evaluate a recent product of this commitment: Advanced Neuroimaging Tools (ANTs), which is approaching its 2.0 release. The ANTs open source software library consists of a suite of state-of-the-art image registration, segmentation and template building tools for quantitative morphometric analysis. In this work, we use ANTs to quantify, for the first time, the impact of similarity metrics on the affine and deformable components of a template-based normalization study. We detail the ANTs implementation of three similarity metrics: squared intensity difference, a new and faster cross-correlation, and voxel-wise mutual information. We then use two-fold cross-validation to compare their performance on openly available, manually labeled, T1-weighted MRI brain image data of 40 subjects (UCLA's LPBA40 dataset). We report evaluation results on cortical and whole brain labels for both the affine and deformable components of the registration. Results indicate that the best ANTs methods are competitive with existing brain extraction results (Jaccard=0.958) and cortical labeling approaches. Mutual information affine mapping combined with cross-correlation diffeomorphic mapping gave the best cortical labeling results (Jaccard=0.669±0.022). Furthermore, our two-fold cross-validation allows us to quantify the similarity of templates derived from different subgroups. Our open code, data and evaluation scripts set performance benchmark parameters for this state-of-the-art toolkit. This is the first study to use a consistent transformation framework to provide a reproducible evaluation of the isolated effect of the similarity metric on optimal template construction and brain labeling.

Figure 1

The evaluation pipeline employs two-fold cross-validation and evaluates the following (affine, deformable) metric pairs: (MSQ,MSQ), (CC,CC), (MI,MI), (MI,MSQ), (MI,CC). The LPBA dataset’s labels provide the ground truth for the subject registration being evaluated.

Figure 2

Two rows of axial slices (neurological convention, i.e. subject left is viewer’s left) taken from each of the templates, constructed from subject group A or B, and by (affine, diffeomorphic) registration according to (MI, MSQ), (MI, CC), or (MI, MI). The top row shows the templates before registration to the (MI, CC) group A template and the bottom row shows them after diffeomorphic registration. The high Jaccard overlaps between these templates’ label sets quantifies and affirms, from an anatomical perspective, the visual similarity in the appearance of the templates. One may see, in the top row, the relative clarity of the MSQ, CC and MI templates. As template acutance does not strictly increase with our performance evaluation outcome, one may conclude that template clarity, alone, is insufficient to determine the neuroanatomical accuracy of an algorithm.

Figure 3

We use the Jaccard overlap metric (intersection of coregistered labeled regions over their union) to compare performance in this evaluation. The data is visualized with a box and whisker plot, with notches. These plots show the median toward the center of the box. The edges of the box delimit the medians of the data above and below the median. The whiskers and points show the minimum and maximum of the data and any points that are plotted may be considered outliers. We used pairwise Student T-tests to determine whether performance differences are significant. In this figure, the MSQAff, CCAff and MIAff overlap results all report the quality of the affine mapping to the derived template from the (MSQ, MSQ), (CC, CC) and (MI, MI) results. Deformable results for these three runs are not shown. The MI-based affine registration gave the best performance for both brain and cortex labeling thus providing the best initialization for follow-up deformable registration. For this reason, the MSQDiff, CCDiff and MIDiff results all use the MI metric for the affine component and MSQ, CC and MI during deformable registration.

Figure 4

A table representing the data used within the box plots of Figure 3 and also showing region-by-region performance for each method. The correlation of the results in the MIAff column with the (MSQ, CC, MI)Diff columns is (0.921, 0.922, 0.944), suggesting the critical role of the affine initialization.

Figure 5

The Jaccard ratios may be similar but show very different errors, as identified visually. We highlight one region of error in the circled region. The Jaccard values are, from left to right, 0.945078, 0.951205 and 0.952316. The (MI,CC)-based Jaccard mean/sd for diffeomorphic brain extraction, over the full dataset, is: 0.958 ± 0.005. This number is determined from taking the mean and standard deviation of the brain extraction overlaps from mapping the group B to the (MI,CC) group A template and vice versa.