Real-Time 3D Tracking of Laparoscopy Training Instruments for Assessment and Feedback

Abstract

Assessment of minimally invasive surgical skills is a non-trivial task, usually requiring the presence and time of expert observers, including subjectivity and requiring special and expensive equipment and software. Although there are virtual simulators that provide self-assessment features, they are limited as the trainee loses the immediate feedback from realistic physical interaction. The physical training boxes, on the other hand, preserve the immediate physical feedback, but lack the automated self-assessment facilities. This study develops an algorithm for real-time tracking of laparoscopy instruments in the video cues of a standard physical laparoscopy training box with a single fisheye camera. The developed visual tracking algorithm recovers the 3D positions of the laparoscopic instrument tips, to which simple colored tapes (markers) are attached. With such system, the extracted instrument trajectories can be digitally processed, and automated self-assessment feedback can be provided. In this way, both the physical interaction feedback would be preserved and the need for the observance of an expert would be overcome. Real-time instrument tracking with a suitable assessment criterion would constitute a significant step towards provision of real-time (immediate) feedback to correct trainee actions and show them how the action should be performed. This study is a step towards achieving this with a low cost, automated, and widely applicable laparoscopy training and assessment system using a standard physical training box equipped with a fisheye camera.

Keywords: cartesian position estimation; laparacospy; laparoscopy training; real-time motion tracking; single view camera; skill metric.

FIGURE 1

(A) Instrument detection, (B) convex Hull area when the rope is crossing the blue marker, (C) convex Hull area when the instrument is crossing the blue marker and obscuring the marker, (D) out of the field of view, when Kalman filter prediction is used (Gautier et al., 2019).

FIGURE 2

Practical workspace for both instruments based on the collected data.

FIGURE 3

Sample ROI squares around the markers.

FIGURE 4

(A) Distance between instruments; (B) crossing between instruments.

FIGURE 5

Representation of the trapezoidal shapes after flattening the view of the cylinders and the angle of the instruments regarding the camera.

FIGURE 6

(A) Estimated α angle, rotation around Z of the instrument around the Center of Rotation and parameters of the trapezoidal detection; (B) parameters of the real instruments; (C) estimated β angle, rotation around X.

FIGURE 7

Diagram of the detection process with image processing.

FIGURE 8

Reference frames at the center of rotation and tool tip (Gautier et al., 2019).

FIGURE 9

The average speed of image processing to detect the corners of a marker on a single instrument in terms of frames-per-second (upper figure) and the rate of correct detection (lower figure) with respect to varying compression rate of video frames transmitted from the camera to the computer.

FIGURE 10

Robotic Surgery Trainer setup used in this study to test the position accuracy of the real-time tracking algorithm.

FIGURE 11

(A) Two sample boxes used to generate the trajectories. (B) The trajectories generated by the robot (red) and tracked by the real-time image processing algorithm (blue) (units: m).

FIGURE 12

Sample laparoscopy instrument tip-point trajectories (units: cm) as successfully discriminated to belong (A) to a professional and (B) to a novice by our novel assessment criterion based on Linear Discriminant Analysis (LDA) presented in (Gautier et al., 2019). The right-hand instrument (the driver) trajectory is in red and the left-hand instrument (the receiver) trajectory is in blue. The LDA line in green shows the best direction to distinguish the right-hand and left-hand instruments according to their spatial distribution and as expected it is almost the same in each case in this specific suturing exercise; reflecting that the orientation of the suturing line is the same and perpendicular to the axis that separates right and left hand tools.