Telepresence for surgical assistance and training using eXtended reality during and after pandemic periods

Abstract

Existing challenges in surgical education (See one, do one, teach one) as well as the COVID-19 pandemic make it necessary to develop new ways for surgical training. Therefore, this work describes the implementation of a scalable remote solution called "TeleSTAR" using immersive, interactive and augmented reality elements which enhances surgical training in the operating room. The system uses a full digital surgical microscope in the context of Ear-Nose-Throat surgery. The microscope is equipped with a modular software augmented reality interface consisting an interactive annotation mode to mark anatomical landmarks using a touch device, an experimental intraoperative image-based stereo-spectral algorithm unit to measure anatomical details and highlight tissue characteristics. The new educational tool was evaluated and tested during the broadcast of three live XR-based three-dimensional cochlear implant surgeries. The system was able to scale to five different remote locations in parallel with low latency and offering a separate two-dimensional YouTube stream with a higher latency. In total more than 150 persons were trained including healthcare professionals, biomedical engineers and medical students.

Keywords: pandemic; Telepresence; annotations; ear–nose–throat; live streaming; mixed reality; surgical training; telehealth.

Figure 1.

Scalable AR-based 3D system design for remote surgical training and education. AR: Augmented Reality; 3D: three-dimensional.

Figure 2.

Audio system design for bi-lateral communication in a remote surgical training environment. (a) Operating surgeon wearing a Bluetooth headset explaining the intervention. (b) Moderating surgeon using a Bluetooth headset and a wireless microphone receiving questions from the remote audience.

Figure 3.

Network configuration: firewall and de-militarized zone.

Figure 4.

Annotation mode of live image. The blue border around the image indicates that the augmentation mode is activated.

Figure 5.

Illumination spectra of the four LED.

Figure 6.

A spectral imaging sequence showing captured 12 spectral images. Relevant peaks in each spectrum are labeled with the corresponding wavelength $\lambda$ position. In total, 10 wavelengths were selected.

Figure 7.

Intermediate calibration showing the color-encoded 2D to 3D correspondence mapping of detected features and 3D model features. Left: Detected features in four different quadrants. Right: Reference 3D model features of rendered model in canonical view. Each feature has a unique ID for a detailed 2D/3D evaluation. 2D: two-dimensional; 3D: three-dimensional.

Figure 8.

Calibration strategy: motor-controlled capturing of different zoom levels.

Figure 9.

Calibration pipeline: motor-controlled capturing of different zoom levels with changing depth of field.

Figure 10.

Timeline in minutes for a cochlear implant at surgery: Timestamps of intraoperative AR-features and annotation tools for remote surgical education. In the Appendix, all procedure steps are described in more details.

Figure 11.

The 12 acquired spectral images of the third patient.

Figure 12.

Comparison of 2D image and corresponding 3D reconstruction. (a) Left view of stereoscopic image pair used for 3D reconstruction. (b) Dense reconstructed point cloud of the surgical scene during a CI insertion. CI: cochlear implant; 2D: two-dimensional; 3D: three-dimensional.

Figure 13.

Professional groups of the registered participants.

Figure 14.

Resulting answers of the questions about features used during the training session (n = xx).

Figure 15.

Results of the questionnaire part about additional possibilities for future training courses (n = xx).

Figure 16.

Results of the questionnaire part about the technology and knowledge of technical solutions.