Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation

Abstract

Major hurdles for Microsoft's HoloLens as a tool in medicine have been accessing tracking data, as well as a relatively high-localisation error of the displayed information; cumulatively resulting in its limited use and minimal quantification. The following work investigates the augmentation of HoloLens with the proprietary image processing SDK Vuforia, allowing integration of data from its front-facing RGB camera to provide more spatially stable holograms for neuronavigational use. Continuous camera tracking was able to maintain hologram registration with a mean perceived drift of 1.41 mm, as well as a mean sub 2-mm surface point localisation accuracy of 53%, all while allowing the researcher to walk about a test area. This represents a 68% improvement for the later and a 34% improvement for the former compared with a typical HoloLens deployment used as a control. Both represent a significant improvement on hologram stability given the current state-of-the-art, and to the best of the authors knowledge are the first example of quantified measurements when augmenting hologram stability using data from the RGB sensor.

Keywords: Microsoft HoloLens augmentation; augmenting hologram stability; cameras; continuous camera tracking; front-facing RGB camera; high-localisation error; hologram registration; holography; image registration; mean perceived drift; medical image processing; minimal quantification; neuronavigation; object tracking; proprietary image processing SDK Vuforia; size 1.41 mm; spatially stable holograms; surface point localisation accuracy; tracking data; typical HoloLens deployment; vuforia tracking.

Fig. 1

Overview of the workflow from DICOM to the hologram

Fig. 2

Sara model. Top row corresponds to (left to right) front, isometric, and right side view. Bottom row contains (left to right) top, bottom, and right side view

Fig. 3

Experimental setup a Illustration of measurement points relative to the phantom b View perspectives of the phantom from each angle of measurement

Fig. 4

Example of measurement techniques a Measuring localisation accuracy by placing the tip of the stylus into the centre of the holographic fiducial b Measuring perceived holographic drift by the difference in similar points. Note: The apparent misalignment between the phantom and the hologram seen in both figures is due to the displacement between the RGB camera used to record the scene and the wearer's line of sight. From the wearer's perspective, the hologram is where it should be

Fig. 5

Control schemes used for manual transforms of models a Translation: allows the user to axially move the model in either the model or world coordinate system using the ‘pinch and drag’ command b Rotation: Allows the user to highlight any axis of the model coordinate system with their gaze and ‘pinch and drag’ to rotate along with it

Fig. 6

Perception of hologram–phantom relationship from a single view a Similarity of an observed hologram relative to its phantom from a single perspective b Two manual registrations highlighting manual registration error. Top row: view from standing; bottom row: view from head on. Variance in vertical height between each registration may be seen