HyperMRI: hyperspectral and magnetic resonance fusion methodology for neurosurgery applications

Abstract

Purpose: Magnetic resonance imaging (MRI) is a common technique in image-guided neurosurgery (IGN). Recent research explores the integration of methods like ultrasound and tomography, among others, with hyperspectral (HS) imaging gaining attention due to its non-invasive real-time tissue classification capabilities. The main challenge is the registration process, often requiring manual intervention. This work introduces an automatic, markerless method for aligning HS images with MRI.

Methods: This work presents a multimodal system that combines RGB-Depth (RGBD) and HS cameras. The RGBD camera captures the patient's facial geometry, which is used for registration with the preoperative MR through ICP. Once MR-depth registration is complete, the integration of HS data is achieved using a calibrated homography transformation. The incorporation of external tracking with a novel calibration method allows camera mobility from the registration position to the craniotomy area. This methodology streamlines the fusion of RGBD, HS and MR images within the craniotomy area.

Results: Using the described system and an anthropomorphic phantom head, the system has been characterised by registering the patient's face in 25 positions and 5 positions resulted in a fiducial registration error of 1.88 ± 0.19 mm and a target registration error of 4.07 ± 1.28 mm, respectively.

Conclusions: This work proposes a new methodology to automatically register MR and HS information with a sufficient accuracy. It can support the neurosurgeons to guide the diagnosis using multimodal data over an augmented reality representation. However, in its preliminary prototype stage, this system exhibits significant promise, driven by its cost-effectiveness and user-friendly design.

Keywords: Computer-assisted intervention; Hyperspectral; Image registration; MRI.

Fig. 1

HyperMRI system

Fig. 2

Homography calibration

Fig. 3

Optitrack-IR calibration

Fig. 4

MRI registration overview: a schematic representation of the MR registration process with depth information. Blue represents data sources, yellow denotes algorithms, purple signifies intermediate results, and in green the final registration output

Fig. 5

MRI registration. First, the patient’s face point cloud is extracted using depth data from HyperMRI; then, this point cloud is registered with the MR volumetric information using ICP; as a result, a fusion of blue (MR) and orange (face point cloud) data and transform G is obtained

Fig. 6

Experiment materials

Fig. 7

Experiment schema for 3D-Slicer and HyperMRI. Orange elements represent landmarks and point clouds used for the registration procedure to extract the FRE metric, while blue elements are used to compute the TRE metric

Fig. 8

AR user interface. a MR (orange) registered with the depth information (grey), and the RGB and HS image planes. b MR (orange) HS image overlapping