Real-time surgical instrument tracking in robot-assisted surgery using multi-domain convolutional neural network

Abstract

Image-based surgical instrument tracking in robot-assisted surgery is an active and challenging research area. Having a real-time knowledge of surgical instrument location is an essential part of a computer-assisted intervention system. Tracking can be used as visual feedback for servo control of a surgical robot or transformed as haptic feedback for surgeon-robot interaction. In this Letter, the authors apply a multi-domain convolutional neural network for fast 2D surgical instrument tracking considering the application for multiple surgical tools and use a focal loss to decrease the effect of easy negative examples. They further introduce a new dataset based on m2cai16-tool and their cadaver experiments due to the lack of established public surgical tool tracking dataset despite significant progress in this field. Their method is evaluated on the introduced dataset and outperforms the state-of-the-art real-time trackers.

Keywords: active research area; challenging research area; computer-assisted intervention system; established public surgical tool tracking dataset; medical robotics; multidomain convolutional neural network; multiple surgical tools; neural nets; real-time knowledge; robot-assisted surgery; surgeon–robot interaction; surgery; surgical instrument location; surgical robot; time surgical instrument tracking.

Fig. 1

Our dataset is made based on cadaver experiment videos for transoral surgery and selected laparoscopic surgery videos in the m2cai16-tool dataset. The corresponding examples of surgical tools are shown here

Fig. 2

Our robot-assisted surgery framework which exploits multi-domain CNN to track surgical tools with endoscopic images as input and surgical tool location as output. The output information with bounding boxes will be further utilised in the processing unit to provide 6D pose estimation, which will provide more benefits for surgical tool navigation

Fig. 3

Precision and success plots using OPE a, b Ablation study: our method compares with RT-MDNet and the corresponding version without instance embedding loss on our STT dataset c–h Show quantitative results of six real-time trackers on m2cai-tool-tracking sub-dataset, robot-assisted-tracking sub-dataset and STT dataset

Fig. 4

Success plots of six real-time trackers over eight tracking challenges a Illumination variation b Background clutter c Deformation d Occlusion e In-plane rotation f Scale variation g Out-of-plane rotation h Motion blur

Fig. 5

Qualitative evaluation of six real-time trackers with example frames shows that our method outperforms the state-of-the-art on STT dataset