Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review

Abstract

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

Keywords: Augmented reality; Head-mounted displays; Human factors; Optical see-through.

Graphical abstract

Fig. 5

Pie chart showing the distribution of the included 91 papers among the identified surgical specialities

Fig. 4

Graphical illustration of included articles grouped by surgical speciality and placed at respective human body regions

Fig. 8

Surgical guidance applications: Distribution of the subset of final 91 articles ($n=59$) by applications of surgical guidance, grouped into the four categories 1. navigation of a linear path, 2. navigation of surgical tools or equipment, 3. navigation of an imaging device, 4. general guidance to help spatial awareness not associated with a specific task

Fig. 9

Guided screw insertion and needle insertion examples. (a) A surgeon uses a custom-made navigation device in an experimental setup (b). Augmented drill entry points (shown in blue) are used to start the navigation. During the guided drill procedure, the 3D angle between current and targeted screw trajectory and their deviation angle are displayed. (source:Liebmann et al. (2019)Fig. 5b and 5d). (c) Mixed reality needle insertion navigation system for low dose-rate (LDR) brachytherapy (source:Zhou et al. (2019b)) Fig. 1.

Fig. 3

Systematic review results overview: Annual Distribution of selected 91 studies from 2013-2020

Fig. 2

Systematic review search strategy

Fig. 10

(a) Robotic instrument placement and endoscopy guidance: Navigation aids for the first assistant: Real-time renderings of a robotic endoscope and robotic instruments that are superimposed on their physical counterparts. In addition, endoscopy guidance is realised via an endoscopy visualisation being registered with a viewing frustrum (source:Fig. 4(f) ofQian et al. (2018)) (b) Robot placement: Reflective-AR Display aided alignment between a real robot arm and its virtual counterpart and subsequent robot placement to its intended position in preparation for robotic surgery (source.Fig. 4ofFotouhi et al. (2020))

Fig. 11

(a) Dissection Guidance example in reconstructive surgery: HoloLens based identification of vascular pedunculated flaps: a CTA-based 3D model of a female patient’s leg consisting of segmented skin, bone, bone, vessels and vascular perforators lower leg is superimposed on the patient anatomy. The surgeon confirms perforator location with audible Doppler ultrasonography (source:Fig. 3ofPratt et al. (2018)) (b) Surgical Anatomy Assessment example in plastic surgery: AR views of the Moverio BT-200 smart glasses showing a patient with osteoma and holographic facial anatomy (face surface and facial bones including the osteoma) superimposed onto a patient’s face (source:Fig. 8ofMitsuno et al. (2017))

Fig. 12

Surgical Training Example Application: Ultrasound Education. (a) Multiple users can see holographic anatomical cross sections mapped on a patient simulator and the ultrasound scan plane. (b) Holograhic subcostal four-chamber view coming out of the simulator probe. Source:Fig. 3and 7 ofMahmood et al. (2018)

Fig. 13

Telementoring Applications. (a) Overview of a Google Glass systeming using a composite surgical field Source:Fig. 3ofPonce et al. (2014). (b) First-person view of HoloLens-based holographic instructions consisting of 3D models and 3D lines Source:Fig. 2ofRojas-Muñoz et al. (2019).

Fig. 14

Surgical Anatomy Assessment and Teleconsultation Applications in Visceral Surgery: (a) Intraoperative visualisation of a preoperative model of the vascular anatomy of the cranio-ventral liver and tumor to be dissected. (b) Intraoperative tele-consulting: real-time video communication with a remote surgeon (Source:Fig. 3(C and F) ofSauer et al. (2017)).

Fig. 15

Distribution of included articles by type of AR visualisation

Fig. 16

Experimental setting, from phantom to animal to clinical studies. Phantom studies dominate and though a number of clinical case studies have been reported (19), we are some way from proving clinical effectiveness of OST-HMDs at present.

Fig. 17

Accuracy verification experiment examples using optical trackers: (a) Accuracy verification block including a metal base with taper holes (for distance and angular error measuring) and 3D-printed cranio-maxillofacial model. (b) A user is conducting the accuracy verification experiment using the accuracy verification block, a tracked calibration tool and tracked OST-HMD (source:Chen et al. (2015)Fig. 7 and 8(c)). Registration accuracy validation using a 3D-printed skull with 10 landmarks (red dots) and a k-wire with attached optical marker (source:Li et al. (2019)Fig. 6).

Fig. 6

Annual Distribution of articles by OST-HMD device from 2014-2020

Fig. 7

Distribution of articles by surgical application context

Fig. 1

Google Scholar search results for surgery “Head Mounted Display” “Augmented Reality” OR “Mixed Reality” surgery “Head Mounted Display” “Augmented Reality” OR “Mixed Reality” “optical see through” OR “Hololens” OR “Magic Leap” OR “Google Glass” in the last 20 years (2020 results have a more recent search date)

Fig. 18

(a) Point-based registration: Human body surface mesh including vessel tree registered to a phantom by landmarks, where surface registration to the HoloLens surface failed (source:Kuhlemann et al. (2017)Fig. 1). (b) Surface registration result: Dummy Head with superimposed 3D CT scan reconstruction of head and intracranial vasculature. HoloLens camera detection of the QR code provides tracking (source:Wu et al. (2018), part of Fig. 9b).

Fig. 19

Registration accuracy verification using a sheet of millimeter paper: (a) Manual and point-based registration: Virtual axes allow the user to translate and rotate a human skull model in order to align it with a phantom. Fiducial markers serving as registration aids are present on both the virtual model and the phantom. (b) Localisation accuracy measurement is realised by placing the tip of a stylus into the center of a holographic fiducial marker. (c) By calculating the difference in similar points the perceived hologram drift is measured. (source:Frantz et al. (2018)Fig. 4a, 4b and 5a).

Fig. 20

Distribution of human factors of conventional non-AR surgical approaches alleviated by the use of OST-HMD AR, grouped into the three categories 1. Information Perception (IP), 2. Cognitive Processing (CP), 3. Control Actions (CA)

Fig. 21

Distribution of persistent human factors of the proposed AR surgical approaches, grouped into the three categories 1. Information Perception (IP), 2. Cognitive Processing (CP), 3. Control Actions (CA)