Virtual and augmented reality for biomedical applications

Abstract

3D visualization technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) have gained popularity in the recent decade. Digital extended reality (XR) technologies have been adopted in various domains ranging from entertainment to education because of their accessibility and affordability. XR modalities create an immersive experience, enabling 3D visualization of the content without a conventional 2D display constraint. Here, we provide a perspective on XR in current biomedical applications and demonstrate case studies using cell biology concepts, multiplexed proteomics images, surgical data for heart operations, and cardiac 3D models. Emerging challenges associated with XR technologies in the context of adverse health effects and a cost comparison of distinct platforms are discussed. The presented XR platforms will be useful for biomedical education, medical training, surgical guidance, and molecular data visualization to enhance trainees' and students' learning, medical operation accuracy, and the comprehensibility of complex biological systems.

Graphical abstract

Figure 1

Extended reality (XR) for biomedical applications and mainstream working principles (A) Virtual reality (VR): visualizing a 3D image of a lung using a head-mounted display (HMD) in VR. (B) Augmented reality (AR): smartphone-based AR. The smartphone augments the brain’s sketch in the real world captured by the camera by overlaying the brain’s virtual image. (C) Mixed reality (MR): visualization of a 3D image of the rib cage using MR glasses. The user can interact with real and virtual objects in the user environment seen through MR glasses. (D) Marker-less tracking in AR. This includes a combination of location data (from Global Positioning System [GPS]), inertial measurement unit (IMU) data (consisting of an accelerometer, gyroscope, and magnetometer), and computer vision (to track image features such as scene depth, the object surface, and object edges). (E) Marker-based tracking in AR. First the smartphone camera captures an image with the scene’s fiducial marker. Then the smartphone’s computer vision system isolates the marker from the scene and removes the back-end background. Next, a virtual coordinate system is drawn with the marker as the reference, and the virtual object is positioned in the scene with respect to the coordinate system. The augmented image is then displayed to the user on the smartphone. (F) Degrees of freedom (DoFs) in VR. VR tracking can have 3 DoFs, which are based on the rotational motion of the user, or 6 DoFs, which consists of rotational and translational movement of the user. (G) Tracking VR principles. Two base stations, placed diagonally across the room, obtain positional data from the HMD and the controllers to track the user’s movement.

Figure 2

VR- and AR-based visualization of scientific experimental imaging data, tools for surgery and anatomy, and collaborative interfaces for education and telehealth (A) Digital whole-slide visualization and navigation using an HMD in VR and a web-based browser for whole-slide imaging on a desktop. (B) Visualization of a user demonstrating a neuron tracing tool. For example, TeraVR can visualize whole-brain imaging data in VR and reconstruct neuron morphology at different regions of interest (ROIs). (C) Visualization and navigation of a 3D scanning electron microscope (SEM) image using VisionVR software by arivis. (D and E) Physicians can use AR to rotate certain anatomy during brain surgery and cardiac surgery to get full visualization to better perform, plan, and explain their surgeries. (F) Studying anatomy using VR can help physicians visualize and explain medical processes to other health professionals. A medical student visualizes multiple organs and organ systems in VR. (G) AR pens can be used to get a 3D image to help students better visualize and study concepts. (H) VR can be used for clinical assessments where the doctor and affected individual can enter a virtual world to receive a checkup.

Figure 3

Case studies using VR and AR (A) Case 1: VR-based visualization of multiplexed protein imaging data. Shown is visualization of highly multiplexed CODEX imaging data (18 markers) obtained from individuals with chronic lymphocytic leukemia (CLL) in ConfocalVR. Slices from each set of markers were converted into RGB stacks of NIfTI (.nii) format. The first image for each condition shows the display control panel. (B) Case 2: AR-based visualization of a cerebral aneurysm for surgical planning. The operator uses the pen as a fiducial to drag the vascular model out of the screen and into AR space. Normal cerebral vessels are gray, the aneurysm is red, and the aneurysm’s neck plane is black. Two snapshots from the real-time operation of this AR tool are presented. (C) Case 4: VR-based Google Cardboard platform for cell biology education. The left side shows the phone screen view, which is a split screen, and the right side shows what can be seen through Google Cardboard, which is a more 3D version of the left side. The images are of a nucleus in a plant cell.