Evaluation of Surgical Skills during Robotic Surgery by Deep Learning-Based Multiple Surgical Instrument Tracking in Training and Actual Operations

Abstract

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons' skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson's correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method.

Keywords: deep learning; quantitative evaluation; robotic surgery; surgical instrument tracking; surgical skills.

Figure 1

Overview of the surgical skill assessment system in robotic surgery. (a) Surgical instrument tracking algorithm. The pipeline consisted of a deep learning-based instance segmentation framework and a tracking framework. Accurate trajectory of the surgical instruments was determined by surgical instrument tip detection and arm-indicator recognition. (b) Assessment of surgical skills. Motion metrics (e.g., instruments out of view) were calculated based on the acquired trajectory of surgical instruments and used to develop a surgical skill assessment system.

Figure 2

Overview of the instance segmentation and tracking frameworks. (a) The instance segmentation framework was trained with three types of training datasets: the bilateral axillo-breast approach (BABA) training model, patients, and a public database. (b) The tracking framework, consisting of a tracker and a sequence of re-identification algorithms. Spatial-temporal re-identification (ST-ReID) was trained with bounding boxes of all types of surgical instruments. Bag of visual words re-identification (BOVW-ReID) was applied after ST-ReID.

Figure 3

Qualitative results of the instance segmentation framework. Recognition of occlusion between surgical instruments located close together or overlapping (red: bipolar (i); pink: bipolar (ii); green: forceps; blue: harmonic; yellow: cautery hook). (a) Application of sample results to the bilateral axillo-breast approach (BABA) training model. (b) Application of sample results to patients.

Figure 4

Trajectory of multi-surgical instrument tip. Each color represents a type of surgical instrument, and the blue area represents the duration of laparoscopy. (a,b) Trajectory of novice surgeons. (c,d) Trajectory of skilled surgeons. (e,f) Trajectory of expert surgeons.

Figure 5

Comparison of the performance of surgical skill prediction models and parts of items in Object Structured Assessment of Technical Skills (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS) with a confusion matrix. The test dataset consisted of four novice, four skilled, and four expert surgeons. (a–c) Confusion matrix results of models using the OSATS. (a) Linear classifier; (b) support vector machine; and (c) random forest. (d–f) Confusion matrix results of models using the GEARS. (d) Linear classifier; (e) support vector machine; and (f) random forest.

Figure 6

Relative importance of motion metrics in surgical skill prediction models. (a) Importance of motion metrics in Object Structured Assessment of Technical Skills (OSATS). (b) Importance of motion metrics in Global Evaluative Assessment of Robotic Surgery (GEARS).