Extended reality for biomedicine

Abstract

Extended reality (XR) refers to an umbrella of methods that allows users to be immersed in a three-dimensional (3D) or a 4D (spatial + temporal) virtual environment to different extents, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). While VR allows a user to be fully immersed in a virtual environment, AR and MR overlay virtual objects over the real physical world. The immersion and interaction of XR provide unparalleled opportunities to extend our world beyond conventional lifestyles. While XR has extensive applications in fields such as entertainment and education, its numerous applications in biomedicine create transformative opportunities in both fundamental research and healthcare. This Primer outlines XR technology from instrumentation to software computation methods, delineating the biomedical applications that have been advanced by state-of-the-art techniques. We further describe the technical advances overcoming current limitations in XR and its applications, providing an entry point for professionals and trainees to thrive in this emerging field.

Figure 1.. Schematic of virtual reality (VR), augmented reality (AR) and mixed reality (MR).

a| The point-of-view for a VR head-mounted display (HMD) allows a user to be fully immersed in a virtual environment. b| AR overlays the virtual donut on top of the real apple regardless of the relative position between two objects. c| MR allows to display the virtual donut partially occluded by the real apple based on the depth information and relative position.

Figure 2.. Instrumentation and optical structure of virtual reality (VR) and augmented reality (AR) head-mounted displays (HMDs).

a| Main hardware components of VR HMDs. b| Main hardware components of AR HMDs. c| The display in a VR HMD projects virtual objects to eyes through optical lenses. d| The optical combiner in an AR HMD merges the real-world scene with virtual objects projected by lenses and the display. e| Field of view (FOV) is defined as the visual field as one eye is relatively stationary, and the edge of a well-designed FOV should be equal to the display screen border.

Figure 3.. Tracking and haptic feedback in extended reality (XR) applications.

a| Inside-out tracking. The sensors such as cameras are mounted on the head-mounted display (HMD) to detect the changes in surroundings with or without markers. b| Outside-in tracking. The sensors are mounted in the stationary location and the markers to be tracked are placed on the target such as HMDs. c| Haptic feedback. Hand gestures are recognized and tracked by sensors for virtual hands display. The collision between virtual hands and virtual objects are detected for the force feedback calculation. The calculated force feedback is delivered through the sensors on the haptic gloves.

Figure 4.. Biomedical applications of extended reality (XR).

a| vLUME facilitates the 3D virtual reality (VR) visualization of millions of molecules, demonstrated by the super-resolved membrane of the T cell . Uses can easily select and isolate complex biological features at the nanoscale. b| The head-fixed mouse placed on a cylindrical styrofoam treadmill is surrounded in a VR environment . Dynamic virtual scenes are created to provide the mouse with the illusion of movement for the investigation of the dopamine circuit activity at various stages. c| The ĒLVIS pipeline allows users to navigate through the real-time diagnostic mapping information on the electroanatomic system . d| Live 3D holograms are created from live transesophageal echocardiography or rotational angiography for the user-directed interaction and manipulation . e| A VR and brain-computer-interface-based training platform induces movement illusion for severe stroke patients, providing patient-driven action observation in head-mounted VR .