Keypoint Transfer Segmentation

Abstract

We present an image segmentation method that transfers label maps of entire organs from the training images to the novel image to be segmented. The transfer is based on sparse correspondences between keypoints that represent automatically identified distinctive image locations. Our segmentation algorithm consists of three steps: (i) keypoint matching, (ii) voting-based keypoint labeling, and (iii) keypoint-based probabilistic transfer of organ label maps. We introduce generative models for the inference of keypoint labels and for image segmentation, where keypoint matches are treated as a latent random variable and are marginalized out as part of the algorithm. We report segmentation results for abdominal organs in whole-body CT and in contrast-enhanced CT images. The accuracy of our method compares favorably to common multi-atlas segmentation while offering a speed-up of about three orders of magnitude. Furthermore, keypoint transfer requires no training phase or registration to an atlas. The algorithm's robustness enables the segmentation of scans with highly variable field-of-view.

Fig. 1

Illustration of keypoint transfer segmentation. First, keypoints (white circles) in training and test images are matched (arrow). Second, voting assigns an organ label to the test keypoint (r.Kidney). Third, matches from the training images with r.Kidney as labels are transferred to the test image, creating a probabilistic segmentation. We show the manual segmentation for comparison.

Fig. 2

Segmentation accuracy for ten organs on ceCT images for majority voting, locally-weighted voting, and keypoint transfer. Bars indicate the mean Dice and error bars correspond to standard error.

Fig. 3

Segmentation accuracy for ten organs on wbCT images for majority voting, locally-weighted voting, and keypoint transfer. Bars indicate the mean Dice and error bars correspond to standard error.

Fig. 4

Coronal views of example segmentation results for ceCT (left) and wbCT (right) overlaid on the intensity images. Each series reports segmentations in the following order: manual, keypoint transfer, locally-weighted multi-atlas.

Fig. 5

Segmentation accuracy for ten organs on ceCT images with keypoint transfer with the number of training images ranging from 5 to 15. Bars indicate the mean Dice over five test images and error bars correspond to standard error.

Fig. 6

Average runtimes (in minutes) of the segmentation of ten organs in one image with keypoint transfer and multi-atlas label fusion for ceCT and wbCT. The time is displayed on the logarithmic scale.

Fig. 7

Coronal views of scans with limited field-of-view showing the kidneys or the spleen, illustrated for ceCT and wbCT, respectively. Bars indicate the mean Dice and error bars correspond to standard error. ‘Spleen Across’ corresponds to using lung and liver keypoints to transfer spleen segmentations.