AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?

Abstract

Introduction: Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers.

Methods: A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection.

Results: An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification.

Conclusions: Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.

Keywords: Artificial intelligence; Computer vision; Surgical data science; Video-based assessment.

Figure 1:

Figure shows The basic maneuvers of suture throw, knot tie and thread cut being performed by a participant performing a simulated enterotomy repair at the ACS 2019 conference

Figure 2:

Figure shows the workflow architecture of the deep learning network. Frames from the video clips for each of the basic maneuvers are extracted and fed into a convolutional neural network and an LSTM. The architecture is then trained to predict the maneuver based on an input video clip.

Figure 3:

Figure shows a concept diagram of a video segmentation tool that is incorporated with our algorithm. The algorithm is able to identify basic maneuvers and based on the procedure step is able to provide information about any error management or safety measures undertaken from the surgeon.