Artificial intelligence based surgical support for experimental laparoscopic Nissen fundoplication

Abstract

Background: Computer vision (CV), a subset of artificial intelligence (AI), enables deep learning models to detect specific events within digital images or videos. Especially in medical imaging, AI/CV holds significant promise analyzing data from x-rays, CT scans, and MRIs. However, the application of AI/CV to support surgery has progressed more slowly. This study presents the development of the first image-based AI/CV model classifying quality indicators of laparoscopic Nissen fundoplication (LNF).

Materials and methods: Six visible quality indicators (VQIs) for Nissen fundoplication were predefined as parameters to build datasets including correct (360° fundoplication) and incorrect configurations (incomplete, twisted wraps, too long (>four knots), too loose, too long, malpositioning (at/below the gastroesophageal junction). In a porcine model, multiple iterations of each VQI were performed. A total of 57 video sequences were processed, extracting 3,138 images at 0.5-second intervals. These images were annotated corresponding to their respective VQIs. The EfficientNet architecture, a typical deep learning model, was employed to train an ensemble of image classifiers, as well as a multi-class classifier, to distinguish between correct and incorrect Nissen wraps.

Results: The AI/CV models demonstrated strong performance in predicting image-based VQIs for Nissen fundoplication. The individual image classifiers achieved an average F1-Score of 0.9738 ± 0.1699 when adjusted for the optimal Equal Error Rate (EER) as the decision boundary. A similar performance was observed using the multi-class classifier. The results remained robust despite extensive image augmentation. For 3/5 classifiers the results remained identical; detection of incomplete and too loose LNFs showed a slight decline in predictive power.

Conclusion: This experimental study demonstrates that an AI/CV algorithm can effectively detect VQIs in digital images of Nissen fundoplications. This proof of concept does not aim to test clinical Nissen fundoplication, but provides experimental evidence that AI/CV models can be trained to classify various laparoscopic images of surgical configurations. In the future, this concept could be developed into AI based real-time surgical support to enhance surgical outcome and patient safety.

Keywords: EfficientNet; Nissen fundoplication; artificial intelligence (AI); computer vision (CV); visual quality indicators (VQIs).

Figure 1

Representative examples of annotated VQIs. (a) correct configuration, (b) incomplete, (c) twisted, (d) too long, (e) too loose, (f) malpositioning below the GE junction.

Figure 2

The confidence of the model for predicting a twisted wrap during testing; its performance is given in the red box at the bottom of the image (97.8%).

Figure 3

Visualization of the rate of true and false predictions for the ensemble of classifiers across all test samples (non-augmented and augmented data) displaying the average performance when tested on their respective VQI or correct samples. The number of samples was normalized between zero and one to enable inter-classifier comparability.

Figure 4

Confusion matrix for prediction results of the multi-class classifier. Values are normalized between zero and one with respect to the ground truth label of a test sample.