Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration

Abstract

Purpose: Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. In this paper, we propose a method for fully automatic registration using anatomical annotations produced by a neural network.

Methods: Neural networks are trained to simultaneously segment anatomy and identify landmarks in fluoroscopy. Training data are obtained using a computationally intensive, intraoperatively incompatible, 2D/3D registration of the pelvis and each femur. Ground truth 2D segmentation labels and anatomical landmark locations are established using projected 3D annotations. Intraoperative registration couples a traditional intensity-based strategy with annotations inferred by the network and requires no human assistance.

Results: Ground truth segmentation labels and anatomical landmarks were obtained in 366 fluoroscopic images across 6 cadaveric specimens. In a leave-one-subject-out experiment, networks trained on these data obtained mean dice coefficients for left and right hemipelves, left and right femurs of 0.86, 0.87, 0.90, and 0.84, respectively. The mean 2D landmark localization error was 5.0 mm. The pelvis was registered within [Formula: see text] for 86% of the images when using the proposed intraoperative approach with an average runtime of 7 s. In comparison, an intensity-only approach without manual initialization registered the pelvis to [Formula: see text] in 18% of images.

Conclusions: We have created the first accurately annotated, non-synthetic, dataset of hip fluoroscopy. By using these annotations as training data for neural networks, state-of-the-art performance in fluoroscopic segmentation and landmark localization was achieved. Integrating these annotations allows for a robust, fully automatic, and efficient intraoperative registration during fluoroscopic navigation of the hip.

Keywords: 2D/3D registration; Landmark detection; Orthopedics; Semantic segmentation; X-ray navigation.

Fig. 1

Three views of the 6 anatomical structures and 14 landmarks to be annotated in 2D fluoroscopy. All landmarks are bilateral with left (L.) and right (R.) denoted. The L. hemipelvis is shown in green, the R. hemipelvis in red, L. femur in cyan, R. femur in orange, vertebrae in blue, and upper sacrum in yellow. Each landmark is overlaid as a purple sphere.

Fig. 2

Example annotations of four specimens. The top row shows the ground truth segmentation labels for each object overlaid onto the fluoroscopic images, along with the landmark locations as yellow circles. The colors of each object correspond to those from Fig. 1. CNN estimates are shown in the second row, with ground truth landmark locations shown as yellow circles and estimated locations shown as yellow crosshairs (+). Missed detections are indicated by a circle without a corresponding cross. Ground truth heatmaps for the R. MOF, L. ASIS, L. GSN, and L. IOF, in (a), (b), (c), and (d), respectively, are overlaid and shown in the third row. Estimated heatmaps for these landmarks are shown in the bottom row. The heatmap shown in (b) highlights a successful no detection report for L. ASIS.

Fig. 3

A plot of 2D landmark detection accuracy given various thresholds in mm. The bilateral cases for each landmark are combined in this plot.

Fig. 4

Abnormal cases with C-arm poses different from the training dataset. The lower hip is not visible in (a), however 2 landmarks were accurately detected allowing a successful pelvis registration. Excessive pelvic tilt is shown in (b), (c), and (d) shows large magnification. Detections with large errors are highlighted by yellow boxes. In (b) a single landmark, out of five detections, had large error, which allowed the registration strategy to succeed. The C-arm pose of (c) causes the boundary along the left femoral neck to appear similar to that adjacent to the IOF in an AP view, resulting in a detection with large error. Pelvis registration in (d) fails due to the large localization errors in each landmark.

Fig. 5

A visualization of all ground truth projection geometries using the APP as the world coordinate frame. Each sphere represents a position of the X-ray source, each square represents the position of the X-ray detector, and each line connects the X-ray source to the principal point on the detector. Red arrows highlight difficult to see geometries of specimen 6. Most of the poses are contained within a 60° range of C-arm orbital rotations and a 30° range of pelvic tilts.