Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker

Abstract

Purpose: To provide an integrated visualisation of intraoperative ultrasound and endoscopic images to facilitate intraoperative guidance, real-time tracking of the ultrasound probe is required. State-of-the-art methods are suitable for planar targets while most of the laparoscopic ultrasound probes are cylindrical objects. A tracking framework for cylindrical objects with a large work space will improve the usability of the intraoperative ultrasound guidance.

Methods: A hybrid marker design that combines circular dots and chessboard vertices is proposed for facilitating tracking cylindrical tools. The circular dots placed over the curved surface are used for pose estimation. The chessboard vertices are employed to provide additional information for resolving the ambiguous pose problem due to the use of planar model points under a monocular camera. Furthermore, temporal information between consecutive images is considered to minimise tracking failures with real-time computational performance.

Results: Detailed validation confirms that our hybrid marker provides a large working space for different tool sizes (6-14 mm in diameter). The tracking framework allows translational movements between 40 and 185 mm along the depth direction and rotational motion around three local orthogonal axes up to [Formula: see text]. Comparative studies with the current state of the art confirm that our approach outperforms existing methods by providing nearly 100% detection rates and accurate pose estimation with mean errors of 2.8 mm and 0.72[Formula: see text]. The tracking algorithm runs at 20 frames per second for [Formula: see text] image resolution videos.

Conclusion: Experiments show that the proposed hybrid marker can be applied to a wide range of surgical tools with superior detection rates and pose estimation accuracies. Both the qualitative and quantitative results demonstrate that our framework can be used not only for assisting intraoperative ultrasound guidance but also for tracking general surgical tools in MIS.

Keywords: Cylindrical marker; Image guidance; Pose estimation; Surgical tool; Tracking.

Fig. 1

a A top view of the proposed marker for cylindrical object tracking. The hybrid marker consists of circular-dot patterns and chessboard vertices. b A side view of a cylinder showing related parameters to define a marker’s local coordinate frame

Fig. 2

Algorithm workflow of the marker detection algorithm. The binary images are generated using threshold values between 70 and 100 with an interval of 10. The detected dots are circled in red, and the chessboard vertices are labelled in yellow. The first dot in the pattern is circled in blue

Fig. 3

A sequence of images showing how the tracking algorithm reacts when the marker is partially occluded by fake blood. The green outline of the marker indicates successful detection while yellow outline means detection failure and tracking component is used

Fig. 4

Qualitative results of the tracking framework for the proposed marker on various types of surgical tool. A 5-mm monopolar cautery (top row), a da Vinci large needle driver (middle row) and a laparoscopic ultrasound probe (bottom row) are tracked, respectively. The coordinate axis in red, green and blue shows that the pose has been correctly estimated

Fig. 5

An illustration of ambiguous pose removal using chessboard vertices. The red and blue circles indicate projections of the model chessboard vertices using two candidate poses. The two quadrangles represent the projected outline of an ultrasound image on the image. In this example, we can easily see that the pose corresponds to the red is correct

Fig. 6

3D printed rigid body with various sizes for validation (left) and definition of the marker’s local coordinate system (right)

Fig. 7

An housing adaptor that can hold a planar ultrasound probe. With the housing, the planar probe can be tracked by using the proposed marker

Fig. 8

A snapshot of the experiment for detection rate comparison. For each distance group, the same movement has been reproduced to test the detection performance of different markers. The circular-dot marker cannot be detected while the maker is in a very slant view. In the far distance group, we show a scenario where the marker is rotated around its axial axis, so only the other side is visible

Fig. 9

Ultrasound image is fused with laparoscopic image using an inverse realism technique [19]