Interactive Flying Frustums (IFFs): spatially aware surgical data visualization

Abstract

Purpose: As the trend toward minimally invasive and percutaneous interventions continues, the importance of appropriate surgical data visualization becomes more evident. Ineffective interventional data display techniques that yield poor ergonomics that hinder hand-eye coordination, and therefore promote frustration which can compromise on-task performance up to adverse outcome. A very common example of ineffective visualization is monitors attached to the base of mobile C-arm X-ray systems.

Methods: We present a spatially and imaging geometry-aware paradigm for visualization of fluoroscopic images using Interactive Flying Frustums (IFFs) in a mixed reality environment. We exploit the fact that the C-arm imaging geometry can be modeled as a pinhole camera giving rise to an 11-degree-of-freedom view frustum on which the X-ray image can be translated while remaining valid. Visualizing IFFs to the surgeon in an augmented reality environment intuitively unites the virtual 2D X-ray image plane and the real 3D patient anatomy. To achieve this visualization, the surgeon and C-arm are tracked relative to the same coordinate frame using image-based localization and mapping, with the augmented reality environment being delivered to the surgeon via a state-of-the-art optical see-through head-mounted display.

Results: The root-mean-squared error of C-arm source tracking after hand-eye calibration was determined as [Formula: see text] and [Formula: see text] in rotation and translation, respectively. Finally, we demonstrated the application of spatially aware data visualization for internal fixation of pelvic fractures and percutaneous vertebroplasty.

Conclusion: Our spatially aware approach to transmission image visualization effectively unites patient anatomy with X-ray images by enabling spatial image manipulation that abides image formation. Our proof-of-principle findings indicate potential applications for surgical tasks that mostly rely on orientational information such as placing the acetabular component in total hip arthroplasty, making us confident that the proposed augmented reality concept can pave the way for improving surgical performance and visuo-motor coordination in fluoroscopy-guided surgery.

Keywords: Augmented reality; Fluoroscopy; Frustum; Surgical data visualization.

Fig. 1

a Schematic illustration of the proposed spatially aware image visualization of X-ray images on their view frustum. In addition, we show transformation to be estimated dynamically to enable the proposed AR environment. Transformations shown as green arrows are estimated directly while transformations shown in orange are derived. b Demonstrates the use of a single IFF from the current view, and c demonstrates the simultaneous visualization of multiple IFFs from the current and previous views

Fig. 2

a A photograph of the marker used for offline calibration of the system. Its 3D geometry, and in particular the location of the 4 infrared reflective spheres, is precisely known enabling 3D pose retrieval via outside-in optical tracking. b An X-ray image of the same marker with c detected centroids of the spheres. When the marker is stationary, poses extracted from a and c enable calibration of the optical tracker to the C-arm source as described in “System calibration” section

Fig. 3

Illustrations describing the process of calibrating the tracker to the C-arm X-ray source using hand–eye calibration and an external optical navigation system. a An infrared reflective marker is attached to the gantry and calibrated to the X-ray source using a second marker that is imaged by the navigation system and the C-arm simultaneously (Fig. 2). b The C-arm gantry, and therefore, the tracker and the optical marker are moved and corresponding pose pairs in the respective frames of reference are collected that are then used for hand–eye calibration following Tsai et al. [38]

Fig. 4

Multiple views of IFFs are shown in a–c. d, e show the augmentation of the virtual view frustum and the corresponding C-arm images from two views on a pelvis and a spine phantom