Cinematic Rendering in Mixed-Reality Holograms: A New 3D Preoperative Planning Tool in Pediatric Heart Surgery

Abstract

Cinematic rendering (CR) is based on a new algorithm that creates a photo-realistic three-dimensional (3D) picture from cross-sectional images. Previous studies have shown its positive impact on preoperative planning. To date, CR presentation has only been possible on 2D screens which limited natural 3D perception. To depict CR-hearts spatially, we used mixed-reality technology and mapped corresponding hearts as holograms in 3D space. Our aim was to assess the benefits of CR-holograms in the preoperative planning of cardiac surgery. Including 3D prints allowed a direct comparison of two spatially resolved display methods. Twenty-six patients were recruited between February and September 2019. CT or MRI was used to visualize the patient's heart preoperatively. The surgeon was shown the anatomy in cross-sections on a 2D screen, followed by spatial representations as a 3D print and as a high-resolution hologram. The holographic representation was carried out using mixed-reality glasses (HoloLens®). To create the 3D prints, corresponding structures were segmented to create STL files which were printed out of resin. In 22 questions, divided in 5 categories (3D-imaging effect, representation of pathology, structure resolution, cost/benefit ratio, influence on surgery), the surgeons compared each spatial representation with the 2D method, using a five-level Likert scale. The surgical preparation time was assessed by comparing retrospectively matched patient pairs, using a paired t-test. CR-holograms surpassed 2D-monitor imaging in all categories. CR-holograms were superior to 3D prints in all categories (mean Likert scale 4.4 ± 1.0 vs. 3.7 ± 1.3, P < 0.05). Compared to 3D prints it especially improved the depth perception (4.7 ± 0.7 vs. 3.7 ± 1.2) and the representation of the pathology (4.4 ± 0.9 vs. 3.6 ± 1.2). 3D imaging reduced the intraoperative preparation time (n = 24, 59 ± 23 min vs. 73 ± 43 min, P < 0.05). In conclusion, the combination of an extremely photo-realistic presentation via cinematic rendering and the spatial presentation in 3D space via mixed-reality technology allows a previously unattained level of comprehension of anatomy and pathology in preoperative planning.

Keywords: 3D printing; cinematic rendering; congenital heart disease; mixed-reality; pediatric heart surgery; preoperative planning.

Figure 1

Surgeon looking at a CR-hologram of a patient's heart with dextro-transposition of the great arteries using HoloLens®. The figure illustrates an image for which the view of the surgeon through the HoloLens® was combined with a photo in which he is working with the HoloLens®.

Figure 2

Processing steps for generating a 3D-printed model (A–D) vs. a CR-hologram (E): (A) DICOM-data viewed in the 3D Slicer application, marked based on an appropriate threshold; (B) a segmented 3D model; (C) a 3D model as an STL file; (D) a finished printed 3D model; (E) cross-sectional images converted to a CR-hologram.

Figure 3

Distribution (means with 95% confidence intervals) of the Likert scale ratings of CR-holograms over the progress of the study.

Figure 4

Presentation of the scores (means with 95% confidence intervals) separately listed for the five subgroups, divided between surgeries where the main pathology was visible or not using the HoloLens®. (A) Cases with an excellent rating for preparation-time benefit (Likert scale = 5; n = 17). (B) Cases without an excellent rating for preparation-time benefit (Likert scale <5; n = 9).

Figure 5

Intraoperative preparation times of patients with preoperative holographic representation (red) and matched patient pairs (blue) without spatial representation. Ratings are divided between cases rated with and without benefit for preparation by the surgeons. Patients rated with “no benefit” did not differ from the rest either in diagnosis or in kind of surgery (see also Table 1B).

Figure 6

Preoperative imaging of a patient's heart with truncus arteriosus communis from two different perspectives. (A1–A3) A direct comparison of CT on a 2D screen (2D imaging), (B1–B3) cinematically rendered hologram, and (C1–C3) 3D printing. In (A3), the coronaries (arrows) show only a weak contrast, but the 3D view can be demonstrated as a cinematically rendered hologram (B3). A reconstruction of the coronary structure for 3D printing was not possible in this case (C3).