Augmented Reality to Compensate for Navigation Inaccuracies

Abstract

This study aims to report on the capability of microscope-based augmented reality (AR) to evaluate registration and navigation accuracy with extracranial and intracranial landmarks and to elaborate on its opportunities and obstacles in compensation for navigation inaccuracies. In a consecutive single surgeon series of 293 patients, automatic intraoperative computed tomography-based registration was performed delivering a high initial registration accuracy with a mean target registration error of 0.84 ± 0.36 mm. Navigation accuracy is evaluated by overlaying a maximum intensity projection or pre-segmented object outlines within the recent focal plane onto the in situ patient anatomy and compensated for by translational and/or rotational in-plane transformations. Using bony landmarks (85 cases), there was two cases where a mismatch was seen. Cortical vascular structures (242 cases) showed a mismatch in 43 cases and cortex representations (40 cases) revealed two inaccurate cases. In all cases, with detected misalignment, a successful spatial compensation was performed (mean correction: bone (6.27 ± 7.31 mm), vascular (3.00 ± 1.93 mm, 0.38° ± 1.06°), and cortex (5.31 ± 1.57 mm, 1.75° ± 2.47°)) increasing navigation accuracy. AR support allows for intermediate and straightforward monitoring of accuracy, enables compensation of spatial misalignments, and thereby provides additional safety by increasing overall accuracy.

Keywords: AR; augmented reality; brain shift; head-up display; microscope-based navigation; navigation accuracy; navigation update; spatial realignment.

Figure 1

Overall workflow incorporating the application of automatic registration using iCT (prior to or after craniotomy), microscope calibration, AR-based navigation verification, and a potential update prior to the durotomy using bony landmarks and AR-based navigation verification, and a potential update using vascular structures and/or cortex representations depending on data availability.

Figure 2

The AR representation of the cranial patient reference array visualizing matching the standard stainless-steel reference unit (A,B). The 2D and 3D AR representations overlaid onto the radiolucent cranial patient reference array used within the iCT set-up matching the central divot (C), and the spherical marker and calibration mark along the reference array’s arm (D).

Figure 3

After calibration of the microscope, besides HUD-based visualization within the microscope, the Microscope Navigation Element (A) and Cranial Navigation element (B) are displayed on monitors close to the surgical field. The microscope application allows for a visualization of outlined objects (yellow: tumor, blue: precentral gyrus, corticospinal tract) superimposed on the microscope video (A, top), or probe’s eye view, target view or outlined 3D anatomy in relation to the microscope video frame (A, bottom, left to right). In parallel, in the navigation application multimodal fused image sets with outlined structures are visualized in, e.g., axial, coronal, and sagittal view (B).

Figure 4

Evaluation of navigation accuracy using a CT-based MIP projection centered at a burr-hole of a prior biopsy surgery, showing a good match of image and patient data (In parallel view of MIP projection (upper part) and inline views with the recent focus plane (blue line) and the optical axis (dashed blue line) in the bottom part).

Figure 5

Using an intraoperatively conducted CT scan for automatic patient registration after craniotomy for evaluation of navigation accuracy showing a sufficient in-plane match of MIP (A) and patient anatomy (B). (In parallel view of MIP projection and patient anatomy (upper part) and inline views with the recent focus plane (blue line) and the optical axis (dashed blue line) in the bottom part).

Figure 6

Navigation inaccuracy seen in the recent focus plane utilizing a MIP of the intraoperative automatic registration CT image set acquired after craniotomy showing the translational mismatch of MIP and patient anatomy (A) and the match of MIP and patients anatomy in the recent focus plane after manual correction (translation) of the visual misalignment (B). (In parallel view of MIP projection (upper part) and inline views with the recent focus plane (blue line) and the optical axis (dashed blue line) in the bottom part).

Figure 7

Navigation inaccuracy visualized in the recent focus plane utilizing a MIP of a preoperative T1-CE image showing the translational mismatch of MIP and patient anatomy (A,B) and the match of MIP and patients anatomy in the recent focus plane after manual correction (translation) of the visual misalignment (C,D). (In parallel view of MIP projection and patient anatomy (upper part) and inline views with the recent focus plane (blue line) and the optical axis (dashed blue line) in the bottom part).

Figure 8

High spatial navigation accuracy seen in the recent focus plane utilizing a MIP of a preoperative T1-CE image (A,C), and patient anatomy enhanced with segmented vascular structures (blue), tumor outlines (yellow) and precentral gyrus (green) (B,D) in two patient cases, showing the different quality of preoperative imaging data (In parallel view of MIP projection and patient anatomy (upper part) and inline views with the recent focus plane (blue line) and the optical axis (dashed blue line) in the bottom part).

Figure 9

Superimposing the microscope video on the 3D visualization of patient MRI data including the pre-segmented objects (cerebrum and tumor) intuitively relating video frame and 3D anatomy (upper part in A–D), in parallel view of AR-supported microscope view, probe’s eye view and target view (bottom part in A–D from left to right). Moving the focus plane (superimposed microscope video) along the optical axis of the microscope (A–D), the registration quality can be evaluated showing a sufficient match in this case.