Object and anatomical feature recognition in surgical video images based on a convolutional neural network

Abstract

Purpose: Artificial intelligence-enabled techniques can process large amounts of surgical data and may be utilized for clinical decision support to recognize or forecast adverse events in an actual intraoperative scenario. To develop an image-guided navigation technology that will help in surgical education, we explored the performance of a convolutional neural network (CNN)-based computer vision system in detecting intraoperative objects.

Methods: The surgical videos used for annotation were recorded during surgeries conducted in the Department of Surgery of Tokyo Women's Medical University from 2019 to 2020. Abdominal endoscopic images were cut out from manually captured surgical videos. An open-source programming framework for CNN was used to design a model that could recognize and segment objects in real time through IBM Visual Insights. The model was used to detect the GI tract, blood, vessels, uterus, forceps, ports, gauze and clips in the surgical images.

Results: The accuracy, precision and recall of the model were 83%, 80% and 92%, respectively. The mean average precision (mAP), the calculated mean of the precision for each object, was 91%. Among surgical tools, the highest recall and precision of 96.3% and 97.9%, respectively, were achieved for forceps. Among the anatomical structures, the highest recall and precision of 92.9% and 91.3%, respectively, were achieved for the GI tract.

Conclusion: The proposed model could detect objects in operative images with high accuracy, highlighting the possibility of using AI-based object recognition techniques for intraoperative navigation. Real-time object recognition will play a major role in navigation surgery and surgical education.

Keywords: Computer vision; Convolutional neural network; Image-guided navigation technology; Object detection; Surgical education.

Fig. 1

Process of making still images for data labeling. A total of 1070 images were cut out from 9 surgical videos including 2 right colorectomies, 4 rectal surgeries, 2 hernia surgeries and 1 sigmoid resection surgery performed in the Department of Surgery at Tokyo Women’s Medical University. Objects are labeled in these images

Fig. 2

Example images of labeling objects. A total of 8 objects, forceps, GI tract, port, gauze, clip, blood, vessel and uterus, were selected and labeled in the images to create an object recognition model. The left-side images are original, and the right-side images show labeled objects. Each object was surrounded carefully with a line for shape recognition. a GI tract and port are labeled. b GI tract, forceps, gauze and blood are labeled. c Blood, forceps and uterus are labeled. d Clip, forceps and vessel are labeled

Fig. 3

Flow of analysis using IBM Visual Insights. The 8 selected objects were labeled in a total of 1070 images that were cut out for creating an object recognition model. The other 200 images for validation were input into the model to verify whether each object was recognized accurately

Fig. 4

Details of the training model. a Accuracy. b Max iteration. c Ratio. d Weight decay. e Momentum. f Learning rate. g Segmentation. h Mean average precision. i Precision. j Recall. k Intersection over union

Fig. 5

Examples of object detection in surgical images. (5a, Ex. 1) The GI tract and port were recognized accurately. (5a, Ex. 2) The GI tract, forceps and gauze were recognized accurately. (5b, Ex. 3) The GI tract, forceps and clips were recognized accurately. (5b, Ex. 4) The GI tract, forceps, gauze and clips were recognized accurately. (5c, Ex. 5) The forceps, vessel and clips were recognized accurately. (5c, Ex. 6) The GI tract, forceps, gauze and clips were recognized accurately

Fig. 6

Example of false negative detection error when the object present in the image was not detected. (6a, Ex. 1) An example with a mistake. There are 4 forceps in the image, but the 4th one was not identified. (6a, Ex. 2) There are 2 ports in the image, but one of them was not identified (6b, Ex. 3) An example with a mistake. There is a clip in the image, but it is recognized as a part of a forceps. (6b, Ex. 4) There is a whitish vessel in the image, but it was not identified

Fig. 7

Example of false positive detection error when one object was detected as another object. (7a, Ex. 1) All GI tracts were recognized accurately, but an intestinal wall was also identified as a GI tract. (7a, Ex. 2) There is no gauze in the image, but a part of the netlike fat is recognized as gauze. (7b, Ex. 3) Part of the fat is recognized as blood (7b, Ex. 4) The uterus is recognized as the GI tract and vice versa