Development of an Intraoperative Pipeline for Holographic Mixed Reality Visualization During Spinal Fusion Surgery

Abstract

Objective. Holographic mixed reality (HMR) allows for the superimposition of computer-generated virtual objects onto the operator's view of the world. Innovative solutions can be developed to enable the use of this technology during surgery. The authors developed and iteratively optimized a pipeline to construct, visualize, and register intraoperative holographic models of patient landmarks during spinal fusion surgery. Methods. The study was carried out in two phases. In phase 1, the custom intraoperative pipeline to generate patient-specific holographic models was developed over 7 patients. In phase 2, registration accuracy was optimized iteratively for 6 patients in a real-time operative setting. Results. In phase 1, an intraoperative pipeline was successfully employed to generate and deploy patient-specific holographic models. In phase 2, the registration error with the native hand-gesture registration was 20.2 ± 10.8 mm (n = 7 test points). Custom controller-based registration significantly reduced the mean registration error to 4.18 ± 2.83 mm (n = 24 test points, P < .01). Accuracy improved over time (B = -.69, P < .0001) with the final patient achieving a registration error of 2.30 ± .58 mm. Across both phases, the average model generation time was 18.0 ± 6.1 minutes (n = 6) for isolated spinal hardware and 33.8 ± 8.6 minutes (n = 6) for spinal anatomy. Conclusions. A custom pipeline is described for the generation of intraoperative 3D holographic models during spine surgery. Registration accuracy dramatically improved with iterative optimization of the pipeline and technique. While significant improvements and advancements need to be made to enable clinical utility, HMR demonstrates significant potential as the next frontier of intraoperative visualization.

Keywords: 3D visualization; HoloLens™; augmented reality; holographic mixed reality; spine surgery.

Figure 1.

Workflow of 3D model generation from intraoperative O-arm scan.

Figure 2.

Example models for (A, B) bony elements with fiducial markers and (C, D) isolated spinal instrumentation.

Figure 3.

HoloLens hand-gesture registration and controls for Unity. (A) Hand-gesture control. B) Manual registration.

Figure 4.

Button commands on Xbox controller for Unity and intraoperative deployment for controller-based registration.

Figure 5.

Holographic mixed reality registration accuracy. (A) Euclidean distance in stealth space between anatomical and holographic test points across time. (B) Average residual error per patient. (C) Mean error for hand-gesture vs controller-based registration. (D) Linear regression model fit for estimating linear trend in error over time.