Artificial Intelligence for Intraoperative Guidance: Using Semantic Segmentation to Identify Surgical Anatomy During Laparoscopic Cholecystectomy

Abstract

Objective: The aim of this study was to develop and evaluate the performance of artificial intelligence (AI) models that can identify safe and dangerous zones of dissection, and anatomical landmarks during laparoscopic cholecystectomy (LC).

Summary background data: Many adverse events during surgery occur due to errors in visual perception and judgment leading to misinterpretation of anatomy. Deep learning, a subfield of AI, can potentially be used to provide real-time guidance intraoperatively.

Methods: Deep learning models were developed and trained to identify safe (Go) and dangerous (No-Go) zones of dissection, liver, gallbladder, and hepatocystic triangle during LC. Annotations were performed by 4 high-volume surgeons. AI predictions were evaluated using 10-fold cross-validation against annotations by expert surgeons. Primary outcomes were intersection- over-union (IOU) and F1 score (validated spatial correlation indices), and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, ± standard deviation.

Results: AI models were trained on 2627 random frames from 290 LC videos, procured from 37 countries, 136 institutions, and 153 surgeons. Mean IOU, F1 score, accuracy, sensitivity, and specificity for the AI to identify Go zones were 0.53 (±0.24), 0.70 (±0.28), 0.94 (±0.05), 0.69 (±0.20). and 0.94 (±0.03), respectively. For No-Go zones, these metrics were 0.71 (±0.29), 0.83 (±0.31), 0.95 (±0.06), 0.80 (±0.21), and 0.98 (±0.05), respectively. Mean IOU for identification of the liver, gallbladder, and hepatocystic triangle were: 0.86 (±0.12), 0.72 (±0.19), and 0.65 (±0.22), respectively.

Conclusions: AI can be used to identify anatomy within the surgical field. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

Figure 1:

A Pyramid Scene Parsing Network (PSPNet) was used for pixel-wise semantic segmentation. The architecture consists of a deep convolutional neural network (CNN; ResNet50) followed by a multi-scale pyramid pooling module, which aggregates the feature maps from the CNN at four different scales (1×1, 2×2, 3×3 and 6×6) using the extracted frames. Details of the model are described in the Supplemental Digital Content (Appendix 2).

Figure 2:

Model outputs are displayed as overlay segmentations of the target anatomical structure(s). For GoNoGoNet, model outputs are demonstrated as both binary format and topographical heat map. In the binary format example (A), each pixel is predicted to be either part of the Go zone (highlighted) or not part of the Go zone (not highlighted). In the probability heat map example (B), each pixel is denoted as a probability of being part of the Go zone (red region: highest probability; blue region: lowest probability).

Figure 3:

Model predictions for GoNoGoNet. Three separate examples of model predictions for GoNoGoNet compared to original frames are shown, displayed as overlays of Go zone (green overlay) and No-Go zone (red overlay), and probability heat maps for Go and No-Go zones.

Figure 4:

Model predictions for CholeNet. Three separate examples of model predictions for CholeNet compared to original frames are shown, displayed as simultaneous overlays of the liver (green overlay), gallbladder (red overlay) and hepatocystic triangle (purple overlay).