Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy

Abstract

Objective(s): To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG).

Background: Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identification of objects and patterns, such as in autonomous driving.

Methods: Intraoperative video from LSG from an academic institution was annotated by 2 fellowship-trained, board-certified bariatric surgeons. Videos were segmented into the following steps: 1) port placement, 2) liver retraction, 3) liver biopsy, 4) gastrocolic ligament dissection, 5) stapling of the stomach, 6) bagging specimen, and 7) final inspection of staple line. Deep neural networks were used to analyze videos. Accuracy of operative step identification by the AI was determined by comparing to surgeon annotations.

Results: Eighty-eight cases of LSG were analyzed. A random 70% sample of these clips was used to train the AI and 30% to test the AI's performance. Mean concordance correlation coefficient for human annotators was 0.862, suggesting excellent agreement. Mean (±SD) accuracy of the AI in identifying operative steps in the test set was 82% ± 4% with a maximum of 85.6%.

Conclusions: AI can extract quantitative surgical data from video with 85.6% accuracy. This suggests operative video could be used as a quantitative data source for research in intraoperative clinical decision support, risk prediction, or outcomes studies.

Figure 1.

A) Traditional neural network architecture where x represents input and w represents the weight of that input within the network. Within each layer are neurons that contain an activation function that considers the sum of the products x_i*w_i in determining whether or not to “fire” and process further information. Each layer passes information sequentially to the next layer until an output is produced. With each successive layer, information can be lost as it is processed and re-weighted by the network. B) A residual neural network is comprised of residual blocks (grey boxes), where an input x can be processed by a layer with a function F to generate a hidden output h. An identity connection allows the input of that block to bypass F. This allows weak connections via F to have less of an impact on the information that is passed down to subsequent blocks with goal of generating an output y.

Figure 2.

Simplified diagram of long short-term memory (LSTM) neural networks. LSTMs are comprised of cells. In this figure, Cell 2 provides more detail on the inner workings of a cell and will serve as the reference point. Cell 2 can receive input from the data as x₂, from Cell 1’s state as C1, and from the hidden layer output of the prior cell h1. Within Cell 2, these inputs can undergo further processing to generate a new hidden output h2 or be passed through the cell along with data from x2 and h1 with relatively little processing into a new cell state C2. This allows information encountered early to be passed along the network as “memory” to assist in calculations later in the network.

Figure 3.

Simplified graphical representation of the architecture of SleeveNet.