Comparative evaluation of registration algorithms in different brain databases with varying difficulty: results and insights

Abstract

Evaluating various algorithms for the inter-subject registration of brain magnetic resonance images (MRI) is a necessary topic receiving growing attention. Existing studies evaluated image registration algorithms in specific tasks or using specific databases (e.g., only for skull-stripped images, only for single-site images, etc.). Consequently, the choice of registration algorithms seems task- and usage/parameter-dependent. Nevertheless, recent large-scale, often multi-institutional imaging-related studies create the need and raise the question whether some registration algorithms can 1) generally apply to various tasks/databases posing various challenges; 2) perform consistently well, and while doing so, 3) require minimal or ideally no parameter tuning. In seeking answers to this question, we evaluated 12 general-purpose registration algorithms, for their generality, accuracy and robustness. We fixed their parameters at values suggested by algorithm developers as reported in the literature. We tested them in 7 databases/tasks, which present one or more of 4 commonly-encountered challenges: 1) inter-subject anatomical variability in skull-stripped images; 2) intensity homogeneity, noise and large structural differences in raw images; 3) imaging protocol and field-of-view (FOV) differences in multi-site data; and 4) missing correspondences in pathology-bearing images. Totally 7,562 registrations were performed. Registration accuracies were measured by (multi-)expert-annotated landmarks or regions of interest (ROIs). To ensure reproducibility, we used public software tools, public databases (whenever possible), and we fully disclose the parameter settings. We show evaluation results, and discuss the performances in light of algorithms' similarity metrics, transformation models and optimization strategies. We also discuss future directions for the algorithm development and evaluations.

Fig. 1

Four randomly-chosen subjects in the NIREP database (the top two rows) and four randomly-chosen subjects in the LONI-LPBA40 database (the bottom two rows). For each subject, both the intensity image and the expert-annotated ROI image are shown. Different colors represent different ROIs in each database. These two databases were used to evaluate how registration methods perform facing challenges arising from the inter-subject variability (Challenge 1).

Fig. 2

Images and annotations of two randomly chosen subjects from each of the three databases we used to represent Challenge 2 (intensity inhomogeneity, noise and structural differences in raw brain images). (a) From the BrainWeb database. (b) From the IBSR database. (c) From the OASIS database.

Fig. 3

Three-plane view of the intensity images and annotation images from three randomly-chosen subjects in the ADNI database. White color in the annotation images denotes the brain masks, and red denotes hippocampus masks. Blue contours in panel (a) point to the region that exists in one image, but does not exist in other images, due to the FOV differences in multiple imaging institutions. The ADNI database was used to represent Challenge 3 (on top of Challenges 1, 2). (a) A normal control (NC) subject. (b) A mild-cognitive-impairment (MCI) subject. (c) An Alzheimer’s Disease (AD) subject.

Fig. 4

Database to evaluate how registration methods perform facing the challenge arising from the pathology-induced missing correspondences (i.e., Challenge 4). Red arrows point out the regions that contain the cavity (after the resection of the original tumors) and the recurrent tumors. Their correspondences are difficult to find in the normal-appearing template image (second row).

Fig. 5

Measuring registration accuracies in different zones. Panel (a) is the sketch of dividing the whole images into various zones. The solid contour filled with yellow texture denotes the abnormal zone (Zone 1), which contains the post-resection cavity and the recurrent tumor. Zones 2 and 3 are normal-appearing regions immediately close to, and far away from, Zone 1. Zone 4 is the whole brain boundary. The definition of the zones can be found in the main context in Section III-D4. Panel (b) shows landmark/ROI definitions for an example pair of images. Blue contours are expert-defined ROIs in Zone 1. Red crosses are expert-defined landmarks in Zone 2. Yellow crosses are expert-defined landmarks in Zone 3. Green contours are the automatically-computed brain boundaries (through Canny edge detection of the brain masks), to measure the registration accuracy in Zone 4. Please note that the landmark/ROI definitions from a second expert (which are not shown here) may differ. This figure is best viewed in color.

Fig. 6

Depiction of inter-expert and algorithm versus expert landmark errors.

Fig. 7

Box-and-Whisker plots of registration accuracies in the NIREP and LONI-LPBA40 databases, as indicated by the Jaccard overlaps averaged across 32 (in NIREP) or 56 (in LONI-LPBA40) ROIs. This figure shows how registration methods perform facing Challenge 1 (inter-subject variability).

Fig. 8

Box-and-Whisker plots of registration accuracy in the BrainWeb database, as indicated by the Jaccard overlaps averaged across 11 available ROIs. This is Scenario 1 in the testing of registration methods facing Challenge 2 (intensity inhomogeneity, noise and structural differences in raw images).

Fig. 9

Registration accuracy in raw brain images, in the IBSR and OASIS databases, as indicated by the Jaccard overlap (the first row) and 95th percentile Hausdorff Distance (the second row), between the warped and the target brain masks. This is Scenario 2 in the testing of registration methods facing Challenge 2 (intensity inhomogeneity, noise and structural differences in raw images). “prctile” in the title of the second subfigure means “percentile”.

Fig. 10

Jaccard overlaps in the ADNI database, for a) the brain mask (the left three columns); b) the left hippocampus (the middle three columns); and c) the right hippocampus (the right three columns). This figure shows how registration methods perform in a typical multi-site database, where additional challenges arise from the imaging and FOV differences in different imaging institutions (Challenge 3).

Fig. 11

Demons, ANTs and DRAMMS registration results of two pairs of images having large anatomical variations especially in ventricles, mainly due to their different levels of neuro-degeneration. All subjects are from the multi-site ADNI database. Blue arrows point to some typical locations where registration results from three methods differ greatly.

Fig. 12

Landmark errors or the 95th percentile Hausdorff Distance in various zones in the pathology-to-normal subject registrations. In addition to the errors, we have shown the average Jaccard overlap in Zone 1 in this figure. This figure shows how registration methods perform in the presence of pathology-induced missing correspondences (Challenge 4).

Fig. 13

Registration of a brain image with tumor recurrence to a normal brain template by DRAMMS, for a series of slices in the coronal view. This figures shows how the mutual-saliency mechanism (a spatial-varying utilization of voxels) helped DRAMMS in the pathological-to-normal subject registration scenario. Without segmentation, initialization, or prior knowledge, the automatically-calculated mutual-saliency map (d), defined in the target image space, effectively assigned low weights to those regions that correspond to those outlier regions (pointed out by arrows) in the source image (a). This way, the negative impact of outlier regions could be largely reduced; registration was mainly driven by regions that could establish good correspondences. Red arrows point to the post-surgery cavity regions. Blue arrows point to the recurrent tumors.