On-the-fly augmented reality for orthopedic surgery using a multimodal fiducial

Abstract

Fluoroscopic x-ray guidance is a cornerstone for percutaneous orthopedic surgical procedures. However, two-dimensional (2-D) observations of the three-dimensional (3-D) anatomy suffer from the effects of projective simplification. Consequently, many x-ray images from various orientations need to be acquired for the surgeon to accurately assess the spatial relations between the patient's anatomy and the surgical tools. We present an on-the-fly surgical support system that provides guidance using augmented reality and can be used in quasiunprepared operating rooms. The proposed system builds upon a multimodality marker and simultaneous localization and mapping technique to cocalibrate an optical see-through head mounted display to a C-arm fluoroscopy system. Then, annotations on the 2-D x-ray images can be rendered as virtual objects in 3-D providing surgical guidance. We quantitatively evaluate the components of the proposed system and, finally, design a feasibility study on a semianthropomorphic phantom. The accuracy of our system was comparable to the traditional image-guided technique while substantially reducing the number of acquired x-ray images as well as procedure time. Our promising results encourage further research on the interaction between virtual and real objects that we believe will directly benefit the proposed method. Further, we would like to explore the capabilities of our on-the-fly augmented reality support system in a larger study directed toward common orthopedic interventions.

Keywords: fluoroscopy; interventional imaging; registration; surgical guidance.

Fig. 1

Spatial transformations for the on-the-fly AR solution.

Fig. 2

Steps in the creation of the multimodality marker. The 3-D-printed template serves as a housing for the marker and is rigidly attached to a carbon fiber rod such that the marker can be safely introduced into the x-ray field of view.

Fig. 3

Source position of the C-arm shown as a cylinder and virtual lines that arise from annotations in the fluoroscopy image.

Fig. 4

Phantoms used in the user studies assessing the performance of the system in an isolated and a surgery-like scenario in (a) and (b), and in x-ray (c) and (d), respectively.

Fig. 5

Setup and the AR view for the “Guidance Experiment Using the AR Environment.”

Fig. 6

X-ray of the second experiment on a semianthropomorphic femur phantom with the K-wire in the final position for one of the participants.