A System for Mixed-Reality Holographic Overlays of Real-Time Rendered 3D-Reconstructed Imaging Using a Video Pass-through Head-Mounted Display-A Pathway to Future Navigation in Chest Wall Surgery

Abstract

Background: Three-dimensional reconstructions of state-of-the-art high-resolution imaging are progressively being used more for preprocedural assessment in thoracic surgery. It is a promising tool that aims to improve patient-specific treatment planning, for example, for minimally invasive or robotic-assisted lung resections. Increasingly available mixed-reality hardware based on video pass-through technology enables the projection of image data as a hologram onto the patient. We describe the novel method of real-time 3D surgical planning in a mixed-reality setting by presenting three representative cases utilizing volume rendering. Materials: A mixed-reality system was set up using a high-performance workstation running a video pass-through-based head-mounted display. Image data from computer tomography were imported and volume-rendered in real-time to be customized through live editing. The image-based hologram was projected onto the patient, highlighting the regions of interest. Results: Three oncological cases were selected to explore the potentials of the mixed-reality system. Two of them presented large tumor masses in the thoracic cavity, while a third case presented an unclear lesion of the chest wall. We aligned real-time rendered 3D holographic image data onto the patient allowing us to investigate the relationship between anatomical structures and their respective body position. Conclusions: The exploration of holographic overlay has proven to be promising in improving preprocedural surgical planning, particularly for complex oncological tasks in the thoracic surgical field. Further studies on outcome-related surgical planning and navigation should therefore be conducted. Ongoing technological progress of extended reality hardware and intelligent software features will most likely enhance applicability and the range of use in surgical fields within the near future.

Keywords: augmented reality; head-mounted display; hologram; image reconstruction; image-guided surgery; mixed reality; navigation; surgical planning; thoracic surgery; volume rendering.

Figure 1

Flowchart on image data and visualization processing.

Figure 2

Illustration of mixed-reality examination setup with image overlay.

Figure 3

Alignment and windowing of high-resolution reconstructed image on patient with large epithelioid sarcoma. The anterior side of the rendered CT image is labeled Ⓐ.

Figure 4

Slicing through overlayed 3D image to examine tumor mass of a solitary fibrous tumor in the right thorax. The right side of the rendered CT image is labeled Ⓡ.

Figure 5

Overlayed rib cage with virtual markings of a costal lesion. The proximal ridge (green arrows) and the distal ridge (purple arrow) have been labeled during case assessment.