A framework for the recognition of high-level surgical tasks from video images for cataract surgeries

Abstract

The need for a better integration of the new generation of computer-assisted-surgical systems has been recently emphasized. One necessity to achieve this objective is to retrieve data from the operating room (OR) with different sensors, then to derive models from these data. Recently, the use of videos from cameras in the OR has demonstrated its efficiency. In this paper, we propose a framework to assist in the development of systems for the automatic recognition of high-level surgical tasks using microscope videos analysis. We validated its use on cataract procedures. The idea is to combine state-of-the-art computer vision techniques with time series analysis. The first step of the framework consisted in the definition of several visual cues for extracting semantic information, therefore, characterizing each frame of the video. Five different pieces of image-based classifiers were, therefore, implemented. A step of pupil segmentation was also applied for dedicated visual cue detection. Time series classification algorithms were then applied to model time-varying data. Dynamic time warping and hidden Markov models were tested. This association combined the advantages of all methods for better understanding of the problem. The framework was finally validated through various studies. Six binary visual cues were chosen along with 12 phases to detect, obtaining accuracies of 94%.

Fig. 1

Framework of the recognition system.

Fig. 2

Different steps of the pupil segmentation. a) input image, b) 1^st step: creation of the mask, c) 2^nd step: Hough transform computation, d) 3^rd step: final segmentation of the pupil.

Fig. 3

Different stages of the segmentation step for the detection of instruments a) the input image, b) the clean mask, c) the region of interest corresponding to the first connected component, d) the ROI corresponding to the second connected component.

Fig. 4

SURF features detected on image from Fig. 3-a., and shown as blue circles. a) SURF points on the first connected component, b) SURF points on the second connected component.

Fig. 5

Left-right HMM, where each state corresponds to one surgical phase

Fig. 6

Typical digital microscope frames for the 12 surgical phases: 1-preparation, 2-betadine injection, 3-lateral corneal incision, 4-principal corneal incision, 5-viscoelastic injection, 6-capsulorhexis, 7-phacoemulsification, 8-cortical aspiration of the big pieces of the lens, 9- cortical aspiration of the remanescent lens, 10-expansion of the principal incision, 11-implantation of the artificial IOL, 12- adjustment of the IOL+ wound sealing

Fig. 7

BVW validation studies comparison of accuracies with different number of visual words and different keypoints detectors: a) Detection of the instruments presence, b) Recognition of the cataract aspect.

Fig. 8

Distance map of two surgeries and dedicated warping path using the Itakura constraint (up), along with transposition of the surgical phases (middle), and the visual cues detected by the system (down).