Value of the surgeon's sightline on hologram registration and targeting in mixed reality

Abstract

Purpose: Mixed reality (MR) is being evaluated as a visual tool for surgical navigation. Current literature presents unclear results on intraoperative accuracy using the Microsoft HoloLens 1^®. This study aims to assess the impact of the surgeon's sightline in an inside-out marker-based MR navigation system for open surgery.

Methods: Surgeons at Akershus University Hospital tested this system. A custom-made phantom was used, containing 18 wire target crosses within its inner walls. A CT scan was obtained in order to segment all wire targets into a single 3D-model (hologram). An in-house software application (CTrue), developed for the Microsoft HoloLens 1, uploaded 3D-models and automatically registered the 3D-model with the phantom. Based on the surgeon's sightline while registering and targeting (free sightline /F/or a strictly perpendicular sightline /P/), 4 scenarios were developed (FF-PF-FP-PP). Target error distance (TED) was obtained in three different working axes-(XYZ).

Results: Six surgeons (5 males, age 29-62) were enrolled. A total of 864 measurements were collected in 4 scenarios, twice. Scenario PP showed the smallest TED in XYZ-axes mean = 2.98 mm ± SD 1.33; 2.28 mm ± SD 1.45; 2.78 mm ± SD 1.91, respectively. Scenario FF showed the largest TED in XYZ-axes with mean = 10.03 mm ± SD 3.19; 6.36 mm ± SD 3.36; 16.11 mm ± SD 8.91, respectively. Multiple comparison tests, grouped in scenarios and axes, showed that the majority of scenario comparisons had significantly different TED values (p < 0.05). Y-axis always presented the smallest TED regardless of scenario tested.

Conclusion: A strictly perpendicular working sightline in relation to the 3D-model achieves the best accuracy results. Shortcomings in this technology, as an intraoperative visual cue, can be overcome by sightline correction. Incidentally, this is the preferred working angle for open surgery.

Keywords: 3D modeling; Augmented reality; Colorectal surgery; Computer-assisted surgery; Image-guided surgery; Three-dimensional imaging.

Fig. 1

Mixed reality navigator for open surgery “MRNOS” phantom. a Angle picture of phantom with XYZ-axes visible. ArUco markers on top of phantom. b Represents the CT-coordinate system used in the phantom. X-axis: right to left, Y-axis: anterior to posterior and Z-axis: superior to inferior, also known as the RAS/LPI nomenclature

Fig. 2

Overview of experimental workflow. a Anatomical internal targets. Wire crosses attached at the inner walls of the MRNOS. b Wire targets visible on CT scan dataset. c 3D model “Hologram” of corresponding wire crosses visible from the outside of the MRNOS. d Needle pins penetrated the 0.8 mm foam to meet the target with the guidance of the hologram. e View of the needle and the target from the inner wall of the MRNOS. Digital Caliper visible. f Hologram automatically registered/embedded into the MROS before the targeting task

Fig. 3

The surgeon’s sightline. Surgeon wearing the HoloLens and facing the MRNOS. a Represents the non-perpendicular free sightline/F/. b Represents the strictly perpendicular sightline/P/. Black and white effect to emphasize the subject and the phantom

Fig. 4

Violin plot showing the probability density of all datapoints. Target error distance (TED) in millimeters (mm). Data grouped in both scenarios (color-coded) and axes (XYZ). Dots represent datapoints

Fig. 5

Violin plot of data divided between attempt 1 and 2. Target error distance (TED) in millimeters (mm). a Values in X-axis grouped in 4 different scenarios. b Values in Y-axis grouped in 4 different scenarios. c Values in Z-axis grouped in 4 different scenarios. Asterisk represents statistically significant differences between attempt 1 and 2

Fig. 6

Graphical simulation of clinical application. Surgeon wears the Microsoft HoloLens 1 and uses the CTrue app, which projects a CT-derived 3D vascular model into the mesentery for open surgery navigation. Red arrow represents the surgeon’s perpendicular sightline