Applying Modern Virtual and Augmented Reality Technologies to Medical Images and Models

Abstract

Recent technological innovations have created new opportunities for the increased adoption of virtual reality (VR) and augmented reality (AR) applications in medicine. While medical applications of VR have historically seen greater adoption from patient-as-user applications, the new era of VR/AR technology has created the conditions for wider adoption of clinician-as-user applications. Historically, adoption to clinical use has been limited in part by the ability of the technology to achieve a sufficient quality of experience. This article reviews the definitions of virtual and augmented reality and briefly covers the history of their development. Currently available options for consumer-level virtual and augmented reality systems are presented, along with a discussion of technical considerations for their adoption in the clinical environment. Finally, a brief review of the literature of medical VR/AR applications is presented prior to introducing a comprehensive conceptual framework for the viewing and manipulation of medical images in virtual and augmented reality. Using this framework, we outline considerations for placing these methods directly into a radiology-based workflow and show how it can be applied to a variety of clinical scenarios.

Keywords: Augmented reality; Radiology; Virtual reality; Visualization.

Fig. 1

An illustration of how the user visualizes virtual elements and real-world images with (a) virtual reality, (b) pass-through augmented reality, and (c) see-through augmented reality headsets

Fig. 2

Illustration of the reality-virtuality continuum. Augmented reality (AR) is a subset of the mixed-reality space between the two extremes of the contiuum, with reality (i.e., a completely real environment) on one end and virtual reality (i.e., a completely virutal environment) on the other

Fig. 3

Delineation of virtual reality experiences as a function of technological sophistication

Fig. 4

A delineation of medical virtual reality applications by the amount of patient involvement including the classification of clinician-as-user and patient-as-user

Fig. 5

Possible workflows for viewing radiology images. The images can be viewed as segmented models or as unsegmented image data with increasing three-dimensional integration

Fig. 6

Radiological image visualization modes using the same scan set. (a) An example of a segmented model derived from a CT scan. (b) An example of a 2D visualization of an unsegmented image. (c) An example of a “2.5D” visualization. (d) An example of (3D) volume rendering

Fig. 7

Flat vs smooth shading. (a), (b) Illustrations of the normal vectors for flat and smooth shading, respectively. (c), (d) The result of using flat and smooth shading, respectively, on a heart model

Fig. 8

Example handheld cutting plane used to visualize the interior chambers of a heart model

Fig. 9

Illustration of back-face rendering options. (a) Unclipped object. (b) Clipped object with back-face culling; when the front corner is clipped, the white background is seen through the opening. (c) Clipped object with back-faces rendered; when the corner is clipped, the inside aspect of the faces in the back of the object is revealed, but the resulting visualization may be confusing to some users. (d) A clipped object with a single, unlit color used to render all back faces; when the corner is clipped, the internal aspect of the faces in the back is now rendered as in (c), but the single color gives the illusion of a solid object

Fig. 10

Back-face rendering options illustrated on a heart model. (a) Back-face culling. (b) Rendering lit back faces. (c) Rendering a single unlit color on back faces. (d) Rendering a single, well-chosen color on back faces

Fig. 11

Illustration of the considerations for apparent resolution for (a) a traditional radiological display and (b) virtual reality visualization. The effect of distance and scale in a virtual reality visualization is illustrated in (c). In all cases shown, the medical image is composed of a fixed number of pixels as shown by the gray grid on top of the heart. As shown in (c), depending on how far or close the user is to the heart in the virtual environment, the number of pixels used to render the heart is different (the vertical screen is always near the eyes)

Fig. 12

Examples of user interactions with medical datasets in virtual reality