Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study

Abstract

Purpose: We explore and validate objective surgeon performance metrics using a novel recorder ("dVLogger") to directly capture surgeon manipulations on the da Vinci® Surgical System. We present the initial construct and concurrent validation study of objective metrics during preselected steps of robot-assisted radical prostatectomy.

Materials and methods: Kinematic and events data were recorded for expert (100 or more cases) and novice (less than 100 cases) surgeons performing bladder mobilization, seminal vesicle dissection, anterior vesicourethral anastomosis and right pelvic lymphadenectomy. Expert/novice metrics were compared using mixed effect statistical modeling (construct validation). Expert reviewers blindly rated seminal vesicle dissection and anterior vesicourethral anastomosis using GEARS (Global Evaluative Assessment of Robotic Skills). Intraclass correlation measured inter-rater variability. Objective metrics were correlated to corresponding GEARS metrics using Spearman's test (concurrent validation).

Results: The performance of 10 experts (mean 810 cases, range 100 to 2,000) and 10 novices (mean 35 cases, range 5 to 80) was evaluated in 100 robot-assisted radical prostatectomy cases. For construct validation the experts completed operative steps faster (p <0.001) with less instrument travel distance (p <0.01), less aggregate instrument idle time (p <0.001), shorter camera path length (p <0.001) and more frequent camera movements (p <0.03). Experts had a greater ratio of dominant-to-nondominant instrument path distance for all steps (p <0.04) except anterior vesicourethral anastomosis. For concurrent validation the median experience of 3 expert reviewers was 300 cases (range 200 to 500). Intraclass correlation among reviewers was 0.6-0.7. For anterior vesicourethral anastomosis and seminal vesicle dissection, kinematic metrics had low associations with GEARS metrics.

Conclusions: Objective metrics revealed experts to be more efficient and directed during preselected steps of robot-assisted radical prostatectomy. Objective metrics had limited associations to GEARS. These findings lay the foundation for developing standardized metrics for surgeon training and assessment.

Keywords: education; prostatic neoplasms; robotic surgical procedures.

Figure 1.

dVLogger, device that records synchronized endoscopic video and system data (kinematic data events) directly from da Vinci Surgical System.

Figure 2.

Scatter plot of performance based on prior experience for 4 study steps of RARP. a, total task completion time as function of prior surgeon experience (console cases). b, instrument path distance of all robotic arms excluding camera. c, average instrument velocity of dominant instrument. Number of cases needed to improve by 50% or 75% in previously mentioned metrics did not differ statistically among 4 steps (p >0.05).

Figure 3.

Illustrative construct validation, with expert and novice surgeon 3-D path tracings of dominant instrument, nondominant instrument and camera during 4 steps of RARP (endoscopic perspective).

Figure 4.

Ratio of nondominant-to-dominant hand use for instrument path distance, instrument moving time and average instrument velocity. Asterisk indicates p <0.05. During dissection tasks (BM, SVD, RLD) expert surgeons used dominant hand more compared to novices. There was no difference in degree of bimanual dexterity during AA.