Neuro-oncological augmented reality planning for intracranial tumor resection

Abstract

Background: Before starting surgery for the resection of an intracranial tumor, its outlines are typically marked on the skin of the patient. This allows for the planning of the optimal skin incision, craniotomy, and angle of approach. Conventionally, the surgeon determines tumor borders using neuronavigation with a tracked pointer. However, interpretation errors can lead to important deviations, especially for deep-seated tumors, potentially resulting in a suboptimal approach with incomplete exposure. Augmented reality (AR) allows displaying of the tumor and critical structures directly on the patient, which can simplify and improve surgical preparation.

Methods: We developed an AR-based workflow for intracranial tumor resection planning deployed on the Microsoft HoloLens II, which exploits the built-in infrared-camera for tracking the patient. We initially performed a phantom study to assess the accuracy of the registration and tracking. Following this, we evaluated the AR-based planning step in a prospective clinical study for patients undergoing resection of a brain tumor. This planning step was performed by 12 surgeons and trainees with varying degrees of experience. After patient registration, tumor outlines were marked on the patient's skin by different investigators, consecutively using a conventional neuronavigation system and an AR-based system. Their performance in both registration and delineation was measured in terms of accuracy and duration and compared.

Results: During phantom testing, registration errors remained below 2.0 mm and 2.0° for both AR-based navigation and conventional neuronavigation, with no significant difference between both systems. In the prospective clinical trial, 20 patients underwent tumor resection planning. Registration accuracy was independent of user experience for both AR-based navigation and the commercial neuronavigation system. AR-guided tumor delineation was deemed superior in 65% of cases, equally good in 30% of cases, and inferior in 5% of cases when compared to the conventional navigation system. The overall planning time (AR = 119 ± 44 s, conventional = 187 ± 56 s) was significantly reduced through the adoption of the AR workflow (p < 0.001), with an average time reduction of 39%.

Conclusion: By providing a more intuitive visualization of relevant data to the surgeon, AR navigation provides an accurate method for tumor resection planning that is quicker and more intuitive than conventional neuronavigation. Further research should focus on intraoperative implementations.

Keywords: augmented reality; brain tumor; computer-assisted surgery; intracranial tumor; neuronavigation; preoperative preparation; resection planning.

Figure 1

Demonstration of the importance of the angle of approach when using pointer-based neuronavigation for (A) superficial as opposed to (B) deep-seated tumors. The green line indicates the correct angulation of the stylus for the chosen entry point, while the red line shows how for the same entry point, improper angulation can lead to important differences in trajectory and offset to the tumor, especially for deep-seated tumors [Created with BioRender.com].

Figure 2

(A) Investigator wearing the AR headset and performing the planning step with the AR application for guidance, using an in-house designed handheld stylus and a stationary reference star, adopted from the Brainlab Curve neuronavigation system. (B) View from within the AR headset, as seen by the investigator at the moment after patient registration and before tumor projection and delineation. The AR overlay shows the 3D tumor and ventricle models in their correct anatomical position. The black cross marked on the patient's skin indicates the central entry point, as predefined by the experienced neurosurgeon. The RGB axes visible on the Brainlab reference star indicate the established 3D cartesian coordinate system in which the patient's position is being tracked. The RGB axes visible on the handheld stylus indicate its correct tracking, along with the white dot aligned on its tooltip.

Figure 3

(A) Segmentation of phantom CT data used for the creation of (B) the target surface-model during pre-clinical registration validation. Note: the enlarged region in the left image indicates a minor surface bias ( ≤ 1 mm) in all dimensions of the surface model (green outline). This bias was discovered in retrospective analysis and could be due to thresholding and partial volume effect of the CT scan images.

Figure 4

Registration pipeline: (A) unregistered phantom; (B) refining initial alignment; (C) registered phantom.

Figure 5

Flow diagram of the different experimental phases for a single case.

Figure 6

Example of the sequence that the investigators followed within the AR application. (A) Patient installed in the Mayfield clamp, with the Brainlab reference star rigidly attached. The RGB axes visible on the Brainlab reference star indicate the established 3D cartesian coordinate system in which the patient's position is being tracked. The virtual skin model is matched to the patient's anatomy during the registration step, in which the boney surfaces along the patient's face are digitized with the handheld stylus, resulting in a point cloud (white dots). (B) After registration, the virtual patient anatomy is displayed in its actual location. Here the cerebrum is shown in pink, with a deep-seated tumor recurrence (located in a previous resection cavity) shown in green. (C) An orthographic projection of the tumor is shown on the patient's skin, allowing delineation of the tumor contours.

Figure 7

Proposed surface-based tumor projection. (A) Wireframe model of the phantom with a deep-seeded tumor (green) projected upon the skin (green outline) nearest to the handheld stylus' tip (red dot). The trajectory between the tumor and stylus is illustrated by the white line, and along with the wireframe skin, it is shown here purely for illustrative purposes and is not visible in the AR application. (B) Example of a clinical case, with the tumor displayed in its actual position as a 3D model inside the patient's head, along with the orthographic projection on the patient's skin, allowing the delineation of the tumor (black marker).

Figure 8

Phantom registration error of AR and conventional neuronavigation techniques shown as an error in rotation and translation. (A) Total registration error magnitudes; (B) AR registration error vectors (RL, right-left; AP, anterior-posterior; SI, superior-inferior; orientations defined in CT space). (C) Conventional neuronavigation registration error vectors. The black diamonds represent outlier data.

Figure 9

Collected point cloud data and registered phantom for a single case; point colors are scaled to point-to-surface distance.

Figure 10

Difference in clinical registration between expert and trainee investigators using (A) the proposed AR workflow and (B) the conventional neuronavigation workflow. The registration variability between expert and non-expert surgeons using the proposed AR method was not statistically different (p = 0.14) when compared to the traditional neuronavigation system. The black diamonds represent outlier data.

Figure 11

Example of the assessment of delineation accuracy: (A) delineation using the AR system; (B) delineation using the conventional neuronavigation system; (C) control based on expert delineation.

Figure 12

Duration of surgical planning phases for each technique. The black diamonds represent outlier data.

Figure 13

Registration of RGB and infrared sensor maps on the HoloLens device, demonstrating the more favorable tracking frustum afforded by the infrared sensor's downturned orientation and wider field-of-view (180°).