The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

Abstract

This is a review focused on advances and current limitations of computer vision (CV) and how CV can help us obtain to more autonomous actions in surgery. It is a follow-up article to one that we previously published in Sensors entitled, "Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?" As opposed to that article that also discussed issues of machine learning, deep learning and natural language processing, this review will delve deeper into the field of CV. Additionally, non-visual forms of data that can aid computerized robots in the performance of more autonomous actions, such as instrument priors and audio haptics, will also be highlighted. Furthermore, the current existential crisis for surgeons, endoscopists and interventional radiologists regarding more autonomy during procedures will be discussed. In summary, this paper will discuss how to harness the power of CV to keep doctors who do interventions in the loop.

Keywords: artificial intelligence surgery; autonomous actions; computer vision; deep learning; machine learning.

Figure 1

PRISMA flow diagram for this review article. * Pubmed ** did not discuss autonomous actions in surgery.

Figure 2

Artificial intelligence (AI) in computerized visualization involves machine learning (ML), which encompasses deep learning (DL). Computer vision (CV) is made possible through the neural networks of DL. Computers and robots may be able to attain autonomous surgical actions through a combination of traditional CV, but also through instrument priors, motion analysis and other non-visual data points.

Figure 3

Computerized visualization in artificial intelligence or autonomous surgery hype cycle with potential steps necessary in-between the peak of inflated expectations, trough of disillusionment, slope of enlightenment and plateau of productivity.

Figure 4

(a) Image of the common bile duct (CBD) (green) during minimally invasive major liver resection and (b) bounded boxes around surgical instruments. Notice that the left and right hepatic ducts are not labeled.

Figure 5

(a) Raw footage of the portal triad during a minimally invasive radical cholecystectomy and common bile duct excision for a patient with gallbladder cancer; (b) bounded boxes showing instance segmentation of the surgical instruments; (c) bounded boxes of instruments and entire arterial supply without multi-class segmentation of the different arteries; (d) patient has a replaced right hepatic artery that is not identified by lower-level segmentation.

Figure 6

(A) Augmented reality (AR) during a minimally invasive extended right liver resection. (B) Organ segmentation is possible with the aid of pre-operatively obtained images. Common bile duct (CBD) in green, right portal vein branch (RPV) in blue, clipped and cut anterior and posterior right hepatic arteries in red.

Figure 7

Real-time augmented reality (AR) for gynecology. Uterus is mobile and deformable (white) and intra-uterine tumors (yellow) (EnCoV, SurgAR, Clermont-Ferrand, France).

Figure 8

Augmented reality (AR) for partial nephrectomy (robotic surgery). Renal arteries (red) and renal tumor (grey-white) (EnCoV, SurgAR, Clermont-Ferrand, France).

Figure 9

Augmented reality (AR) for partial hepatectomy. Intra-hectic portal vein (blue) and hepatic tumors (yellow) (EnCoV, SurgAR, Clermont-Ferrand, France).