Virtual Reality and Augmented Reality in Plastic Surgery: A Review

Abstract

Recently, virtual reality (VR) and augmented reality (AR) have received increasing attention, with the development of VR/AR devices such as head-mounted displays, haptic devices, and AR glasses. Medicine is considered to be one of the most effective applications of VR/AR. In this article, we describe a systematic literature review conducted to investigate the state-of-the-art VR/AR technology relevant to plastic surgery. The 35 studies that were ultimately selected were categorized into 3 representative topics: VR/AR-based preoperative planning, navigation, and training. In addition, future trends of VR/AR technology associated with plastic surgery and related fields are discussed.

Keywords: Augmented reality; Plastic surgery; Virtual reality; Virtual simulation; Virtual surgery.

Fig. 1. VR-based workbench system for mandibular reconstruction surgery

VR-based workbench system as seen from above (A) and from the side (B). The monitor (a) displays the anatomical 3D model, which is reflected on the half-transparent mirror (b). The surgeon manipulates the 3D model with the haptic device (c) under the mirror. The infrared cameras (d) track the markers on the stereo glasses (e) for user look-around. VR, virtual reality; 3D, 3-dimensional. Adapted from Olsson et al. Plast Reconstr Surg Glob Open 2015;3:e479 [8], with permission of Wolters Kluwer Health Inc. via Copyright Clearance Center.

Fig. 2. Reconstruction planning workflow

(A) Load segmented bone and vessels from computed tomography angiogram data and resect bone to prepare the recipient site for reconstruction. (B) Define the positions, orientations, and angulations of fibula segments; test pedicle reach to anastomosis sites on the recipient vessels; and test possible skin paddle configurations. (C) Resulting plan. The user iterates within the Design and Test stage to find a suitable configuration for the fibula, vessels, and skin paddle. Adapted from Olsson et al. Plast Reconstr Surg Glob Open 2015;3:e479 [8], with permission of Wolters Kluwer Health Inc. via Copyright Clearance Center.

Fig. 3. Virtual reconstruction of mandible

Each individual bone fragment is given a unique color (A). When the haptic cursor is held close to a bone fragment, it is highlighted (B) and the user can then grasp and manipulate it with the 6-DOF haptic handle (C). Contact forces guide the user during manipulation. DOF, degree of freedom. Adapted from Olsson et al. Int J Comput Assist Radiol Surg 2013;8:887-94 [10], on the basis of Open Access.

Fig. 4. Wearable AR system for maxillofacial surgery

The two external cameras acquire real video frames. The software application merges the virtual 3D model derived during surgical planning with real data from the camera frames and sends the result to the two internal monitors. Alignment between real and virtual information is obtained by calculating the positions of colored markers relative to camera data, with respect to their known positions (recorded during planning), using detailed preoperative CT images). AR, augmented reality; 3D, 3-dimensional; CT, computed tomography. Adapted from Badiali et al. J Craniomaxillofac Surg 2014;42:1970-6 [19], with permission of Elsevier via Copyright Clearance Center.

Fig. 5. Application of AR-based display in orthognathic surgery

Surgeon's view of the segmented virtual maxilla in an AR environment. AR, augmented reality. Adapted from Mischkowski et al. J Craniomaxillofac Surg 2006;34:478-83 [21], with permission of Elsevier via Copyright Clearance Center.

Fig. 6. Augmented fusion of patient's models for surgical visualization

(A) and (B) Image registration results using the lower front teeth and lower left molars models. Nerve canals are overlaid on the image for surgical visualization. (C) and (D) Image registration result using the upper front teeth model and the resulting augmented fusion of the maxillofacial model with the camera video for surgical visualization. Adapted from Wang et al. Int J Med Robot 2016;2016 Jun 9 [Epub]. http://doi.org/10.1002/rcs.1754 [23], with permission of John Wiley and Sons via Copyright Clearance Center.

Fig. 7. Markerless AR-based system for oral and maxillofacial surgery

(A) Picture of the surgeon wearing the 4K camera. (B) Teeth tracking and (C) video see-through augmented reality validated on clinical data. The model of the carotid artery of the patient is overlaid. Adapted from Wang et al. Int J Med Robot 2016;2016 Jun 9 [Epub]. http://doi.org/10.1002/rcs.1754 [23], with permission of John Wiley and Sons via Copyright Clearance Center.

Fig. 8. Virtual training system for maxillofacial surgery

(A) A surgeon evaluating use of the simulator, and (B) the bone sawing procedure for 6 trials. Adapted from Lin Y, et al. J Biomed Inform 2014;48: 122-9 [27], with permission of Elsevier via Copyright Clearance Center.

Fig. 9. VR-based training for orthopedic surgery

OssoVR, virtual training simulation of orthopedic surgery using virtual reality (VR). Adapted from http://ossovr.com [31], with permission of OssoVR.

Fig. 10. VR-based simulation system for neurosurgery

A general view of the immersive virtual reality (VR) environment being used by a resident. The operator is using haptic devices in both right and left hand which mimic the operating room environment. The virtual image is projected on a screen in front of the resident. Adapted from Alaraj A, et al. Neurosurgery 2015;11 Suppl 2:52-8 [36], on the basis of Open Access.