Merging virtual and physical experiences: extended realities in cardiovascular medicine

Abstract

Technological advancement and the COVID-19 pandemic have brought virtual learning and working into our daily lives. Extended realities (XR), an umbrella term for all the immersive technologies that merge virtual and physical experiences, will undoubtedly be an indispensable part of future clinical practice. The intuitive and three-dimensional nature of XR has great potential to benefit healthcare providers and empower patients and physicians. In the past decade, the implementation of XR into cardiovascular medicine has flourished such that it is now integrated into medical training, patient education, pre-procedural planning, intra-procedural visualization, and post-procedural care. This review article discussed how XR could provide innovative care and complement traditional practice, as well as addressing its limitations and considering its future perspectives.

Keywords: Augmented reality; Extended reality; Imaging; Intervention; Mixed reality; Virtual reality.

Graphical Abstract

Extended reality (XR) is the umbrella term for all immersive technologies that merge virtual and physical experiences. XR has the potential to be integrated into all stages of cardiovascular medicine, including education, telemedicine, procedure planning, intraprocedural visualization, post-procedure care, and for rehabilitation. XR can be applied to all participants in cardiovascular medicine, from patients to trainees to clinicians. However, XR technologies need to overcome the barriers of operation cost, lack of evidence, insufficient patient/physician input in development, and system reliability to achieve its full potential in cardiovascular medicine.

Figure 1

The spectrum of extended realities: the spectrum of XR technologies across the experience provided and image quality is displayed. The left upper square shows the quality of the ideal XR system, which will provide high-definition images with great interactiveness. AR = augmented reality, MeR = merged reality, MxR = mixed reality, VR = virtual reality, XR = extended reality.

Figure 2

Characteristics and applications of XR technologies: the pros and cons and the applications of VR, MeR, and MxR technologies. Based on images from Zlahoda et al., under submission.

Figure 3

Examples of XR devices: the selected examples of the currently available XR devices.

Figure 4

Virtual heart team discussion during the COVID-19 pandemic: the demonstration of virtual heart team discussion using XR technology to achieve simultaneous visualization and manipulation of the patient’s images.

Figure 5

XR-integrated clinical workflow in coronary artery bypass surgery: (A) shows that for patients with coronary artery disease, we can use non-invasive anatomic and functional CTA as the sole guide for planning revascularization. CTA = computed tomography angiogram, FFRct = CT derived functional flow reserve, MIP = maximal intensity projection, MPP = multi-planer projection. (B) shows the XR-integrated heart team discussion with direct graft length measurement in XR and the surgical plan. LIMA = left internal mammary artery, SVG = saphenous vein graft. (C) shows side-by-side the CTA analysis, the 3D hologram for visualization and manipulation during the procedure, and the actual surgical view of a different example case D1 = first diagonal branch, LIMA = left internal mammary artery, OM = obtuse marginal branch, LAD = left anterior descending artery, RA = radial artery graft. Picture courtesy of Dr. Fabio Ramponi and Dr. John Puskas. (D) shows the post-operative review at 30-day follow-up with the anatomical and functional assessment showing that an SVG was inappropriately attached to segment 2. The 3D model shows that the unsatisfactory result could have been avoided by using the RIMA for RCA. A 3D printing model was also created for direct assessment. LIMA = left internal mammary artery, SVG = saphenous vein graft, RIMA = right internal mammary artery.

Figure 5

(Continued)

Figure 6

Three-dimensional model of aortic dissection constructed from CT images showing anatomical details and pressure distribution: panel A shows the detailed anatomical 3D model of the aortic dissection false lumen overlayed on the true lumen. This model shows the entry sites (circle), the re-entry sites (arrows), and the spinal arteries connected to the false lumen (arrowheads). Panel B shows the distribution of pressure at mid-systole (Left) and early diastole (Right).

Figure 7

An example of intraprocedural OCT reconstruction and visualization using MxR technology: panel A shows the post-balloon dilatation OCT run-through where a concerning flying object was noted (arrow). Panel B shows the 3D reconstruction of the OCT images, which revealed that the flying object was, in fact, a floating band. Panel C shows the interventionist’s view of the reconstructed 3D images in MxR with the floating band in sight. OCT = optical coherent tomography.

Figure 8

Intraprocedural use of MxR: panel A shows real-time 3D trans-esophageal echocardiography visualization during an atrial septal defect closure with HoloLens mixed reality goggles. The data was streamed to the operator, who can control the volume rendering independent of the echocardiographer. Panel B shows real-time 3D trans-oesophageal echocardiography visualized on the EchoPixel platform. Courtesy of MedApp (Krakow, Poland).