Robust hand tracking for surgical telestration

Abstract

Purpose: As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs.

Methods: We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching.

Results: According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied.

Conclusion: Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.

Keywords: Computer vision; Deep learning; Hand tracking; Surgical data science; Telestration.

Fig. 1

Our telestration approach compared to the state of the art. a Previous approaches to surgical telestration rely on overlaying drawings on laparoscopic videos, while our concept is based on b the augmented reality (AR) visualization of the expert surgeon’s hand

Fig. 2

Concept overview. Our approach to surgical telestration relies on a camera that continuously captures a hand of the mentor who observes the operation either on-site or remotely. The camera data are processed by a two-stage neural network, which outputs both the skeleton (represented by 21 keypoints) and the segmented hand. The hand segmentation is overlaid on the surgical screen for intuitive coaching, while the skeleton representation is stored for long-term analysis

Fig. 3

Overview of the models used for real-time hand tracking. Our approach comprises three core components. (1) a bounding box module using the YOLOv5s architecture, (2) a skeleton tracking module using an EfficientNet B3 and (3) a segmentation module using a FPN-EfficientNet B1. (2) and (3) operate on images cropped to the respective bounding boxes (see “Real-time hand localization”, “Real-time skeleton tracking”, “Real-time hand segmentation” section)

Fig. 4

Representative results for a diverse set of gestures. The outputs of the three models for bounding box prediction (top) as well as skeleton tracking and segmentation (bottom) are shown

Fig. 5

Skeleton tracking performance for our method (orange) vs. MediaPipe (blue) as the baseline. Fraction of successful localizations (left) and mean regression distance (right) for successful localizations and validated with respect to the different hand properties. Note that for MediaPipe there are only very few successful localizations for blue gloves and none for green ones

Fig. 6

Representative failure cases of the skeleton extraction model (top row) and the segmentation model (bottom row)

Fig. 7

Results of the prospective validation study. Skeleton tracking performance (upper row), quantified by mean regression distance and hand tracking performance (lower row), quantified by the dice similarity coefficient (DSC) are shown for the camera used in the training data set (D435i) as well as a previously unseen camera (L515). Each color corresponds to a different mentor. No notable differences were obtained for the different gestures