Evaluation of Multi-Atlas Label Fusion for In Vivo MRI Orbital Segmentation

Abstract

Multi-atlas methods have been successful for brain segmentation, but their application to smaller anatomies remains relatively unexplored. We evaluate 7 statistical and voting-based label fusion algorithms (and 6 additional variants) to segment the optic nerves, eye globes and chiasm. For non-local STAPLE, we evaluate different intensity similarity measures (including mean square difference, locally normalized cross correlation, and a hybrid approach). Each algorithm is evaluated in terms of the Dice overlap and symmetric surface distance metrics. Finally, we evaluate refinement of label fusion results using a learning based correction method for consistent bias correction and Markov random field regularization. The multi-atlas labeling pipelines were evaluated on a cohort of 35 subjects including both healthy controls and patients. Across all three structures, NLSS with a mixed weighting type provided the most consistent results; for the optic nerve NLSS resulted in a median Dice similarity coefficient of 0.81, mean surface distance of 0.41 mm and Hausdorff distance 2.18 mm for the optic nerves. Joint label fusion resulted in slightly superior median performance for the optic nerves (0.82, 0.39 mm and 2.15 mm), but slightly worse on the globes. The fully automated multi-atlas labeling approach provides robust segmentations of orbital structures on MRI even in patients for whom significant atrophy (optic nerve head drusen) or inflammation (multiple sclerosis) is present.

Keywords: Label Fusion; MRI; Multi-Atlas; Optic Nerve; Segmentation.

Fig. 1

Acquired images (left) and manual labels (right) are shown for two controls (a, b), a multiple sclerosis patient (c), and an optic nerve head drusen patient (d). Axial, sagittal, and coronal views of the manually labeled optic nerves, optic chiasm, and eye globes for the subject in (b) are shown in (e).

Fig. 2

Flowchart of the orbit structures multiatlas segmentation evaluation pipeline and postprocessing. A geometric deformation between an atlas and the target is computed. After registration, the set of labels in the coordinate system of each atlas are transformed to the target space. The propagated labels are fused using the label fusion algorithms in a leave-one-out cross-validation approach. Quantitative accuracy is assessed using the Dice similarity coefficient, Hausdorff distance, and mean surface distance. The symmetric surface distance metrics are computed in both directions in terms of distance from the expert labels to the estimated segmentations and vice versa. To address the problem of systematic outliers, we evaluate Markov random field regularization on the label priors. As an alternative postprocessing step, we consider a method of learning-based voxel-wise correction which constructs classifiers to recognize and correct spatial locations and intensity patterns where mislabeling is most likely to occur. Note that the optional steps are indicated as “opt.”

Fig. 3

Quantitative results of the evaluation of label fusion algorithms on the (a) optic nerves, (b) optic chiasm, and (c) eye globes show that nonlocal spatial simultaneous truth and performance level estimation (STAPLE) with a combination of mean square difference and locally normalized correlation coefficient similarity weighting type is the most consistent performer across the three structures and outperforms nonlocal STAPLE in each case. Although local weighted vote results in high Dice coefficients, it is susceptible to more number of outliers in distance-based error metrics. The situation is similar for joint fusion and the distance errors are high especially in the optic nerves. For the eye globes, which have high contrast, compared to the background, almost similar results are obtained for each method.

Fig. 4

Statistical assessment of the methodological difference in label fusion.

Fig. 5

Qualitative results showing point-wise surface distance error of the label fusion results with respect to expert labels. For a typical subject, the top row depicts surface distance error calculated from the truth to estimate for each fusion method, and the bottom row depicts the same calculated from the estimate to the truth. Overall, voting-based fusion methods result in larger distance errors at the boundaries compared to statistical label fusion methods. The nonlocal fusion methods estimate the shape and boundaries more accurately compared to STAPLE and spatial STAPLE (spSTAPLE). For the nonlocal fusion methods, results for only the mixed weighting type are shown.

Fig. 6

Qualitative results showing the performance of each fusion method for a typical and a poor segmentation. In the latter case, voting methods fail to captures the nerve and chiasm boundaries. Note that the STAPLE and spSTAPLE result in systematically larger (over-segmented) labels than the truth. Nonlocal statistical fusion methods overcome the over segmentation problem and result in more accurate estimation of boundaries and are able to capture the smooth and tapering tubular structures of the nerves. For the nonlocal fusion methods, results for only the mixed weighting type are shown.

Fig. 7

Relative volume difference calculated for each fusion method shows that nonlocal spSTAPLE with a mixed weighting type results in minimum error for all three structures. STAPLE and spSTAPLE result in over segmentation for the nerves and the chiasm. (a) Optic nerves, (b) optic chiasm, and (c) eye globes.

Fig. 8

Variation of median Dice similarity coefficient for increasing a number of atlases (five cross-validation experiments for each set of atlases). Error bars show the standard deviation across repeated measures. The nonlocal fusion methods result in consistent performance compared to others whose performance drops for the case of a few ($<5$) atlases. With increasing number of atlases, voting-based methods perform almost similar to the nonlocal methods. (a) Dice overlap (optic nerves), (b) Dice overlap (optic chiasm), and (c) Dice overlap (eye globes).

Fig. 9

Quantitative evaluation of optic nerve segmentation with MRF regularization and classifier-based error correction on nonlocal spSTAPLE label fusion estimate in terms Dice overlap (a), distance errors (b and c), and relative volume difference (d). Error correction eliminates the outliers in surface distance errors.

Fig. 10

Qualitative comparison of nonlocal spSTAPLE label fusion estimate with and without error corrections. Although it improves the shape, error correction also leads to loss of information at thin connected regions.