Multidisciplinary validation study of the da Vinci Skills Simulator: educational tool and assessment device

Abstract

Despite the increased dexterity and precision of robotic surgery, like any new surgical technology it is still associated with a learning curve that can impact patient outcomes. The use of surgical simulators outside of the operating room, in a low-stakes environment, has been shown to shorten such learning curves. We present a multidisciplinary validation study of a robotic surgery simulator, the da Vinci(®) Skills Simulator (dVSS). Trainees and attending faculty from the University of Toronto, Departments of Surgery and Obstetrics and Gynecology (ObGyn), were recruited to participate in this validation study. All participants completed seven different exercises on the dVSS (Camera Targeting 1, Peg Board 1, Peg Board 2, Ring Walk 2, Match Board 1, Thread the Rings, Suture Sponge 1) and, using the da Vinci S Robot (dVR), completed two standardized skill tasks (Ring Transfer, Needle Passing). Participants were categorized as novice robotic surgeon (NRS) and experienced robotic surgeon (ERS) based on the number of robotic cases performed. Statistical analysis was conducted using independent T test and non-parametric Spearman's correlation. A total of 53 participants were included in the study: 27 urology, 13 ObGyn, and 13 thoracic surgery (Table 1). Most participants (89 %) either had no prior console experience or had performed <10 robotic cases, while one (2 %) had performed 10-20 cases and five (9 %) had performed ≥20 robotic surgeries. The dVSS demonstrated excellent face and content validity and 97 and 86 % of participants agreed that it was useful for residency training and post-graduate training, respectively. The dVSS also demonstrated construct validity, with NRS performing significantly worse than ERS on most exercises with respect to overall score, time to completion, economy of motion, and errors (Table 2). Excellent concurrent validity was also demonstrated as dVSS scores for most exercises correlated with performance of the two standardized skill tasks using the dVR (Table 3). This multidisciplinary validation study of the dVSS provides excellent face, content, construct, and concurrent validity evidence, which supports its integrated use in a comprehensive robotic surgery training program, both as an educational tool and potentially as an assessment device. Table 1 dVSS validation study participant demographic information Survey question Response Number (%) Gender Male 36 (67.9) Female 17 (32.1) Handedness Right-hand dominant 45 (84.9) Left-hand dominant 4 (7.5) Ambidextrous 3 (5.7) Level of training Junior Resident (R1-R3) 17 (32.1) Senior Resident (R4-R5) 12 (22.6) Fellow 16 (30.2) Staff Surgeon 8 (15.1) Specialty Urology 27 (50.9) ObGyn 13 (24.5) Thoracics 13 (24.5) Previous MIS experience (laparoscopic or thoracoscopic) None/minimal 17 (32.1) Moderate 11 (20.8) Significant 18 (34.0) Fellowship-trained in MIS 4 (7.5) Previous robotic surgery experience None 32 (60.4) Yes 21 (39.6) If yes, number of operative cases as surgical assistant 0 cases 33 (62.3) <10 cases 9 (17.0) 10-20 cases 3 (5.7) >20 cases 8 (9.4) If yes, number of operative cases at robotic console for at least 30 min 0 cases 41 (77.4) <10 cases 6 (11.3) 10-20 cases 1 (1.9) >20 cases 5 (9.4) MIS minimally invasive surgery Table 2 dVSS construct validity evidence dVSS exercise All subjects' overall score (%, mean ± SD) Novice robotic surgeon overall score (%, mean ± SD) Expert robotic surgeon overall score (%, mean ± SD) p value Camera Targeting 1 69.943 ± 21.7489 67.170 ± 21.5258 91.667 ± 4.2269 0.008 Peg Board 1 78.596 ± 11.9824 76.913 ± 11.6616 91.500 ± 3.8341 0.004 Match Board 1 69.880 ± 17.7691 67.864 ± 17.9075 84.667 ± 6.1860 0.028 Thread the Rings 74.152 ± 16.4289 71.825 ± 16.2605 89.667 ± 5.8878 0.011 Suture Sponge 1 74.787 ± 14.3086 73.171 ± 14.5067 85.833 ± 5.6716 0.042 Ring Walk 2 75.098 ± 20.0861 73.333 ± 20.1099 88.333 ± 15.4100 0.086 Peg Board 2 84.308 ± 11.7633 83.283 ± 12.0861 92.167 ± 3.6009 0.082 Table 3 dVSS concurrent validity evidence NP time NP errors RT time RT errors Camera Targeting 1 overall score 0.471 (0.001) 0.083 (0.575) 0.291 (0.045) 0.061 (0.685) Peg Board 1 overall score 0.486 (0.001) 0.141 (0.344) 0.325 (0.026) 0.088 (0.555) Match Board 1 overall score 0.543 (<0.001) 0.096 (0.530) 0.295 (0.050) 0.215 (0.162) Thread the Rings overall score 0.432 (0.005) 0.231 (0.147) 0.533 (<0.001) 0.163 (0.310) Suture Sponge 1 overall score 0.592 (<0.001) 0.105 (0.509) 0.437 (0.004) 0.015 (0.925) Ring Walk 2 overall score 0.454 (0.002) 0.179 (0.234) 0.399 (0.006) 0.022 (0.884) Peg Board 2 overall score 0.675 (<0.001) 0.058 (0.696) 0.073 (0.626) 0.045 (0.762) Subjects' overall score for each dVSS exercise is correlated with the time to complete (time) and number of errors (errors) for the Needle Passing (NP) and Ring Transfer (RT) tasks performed using the dVR. Data is expressed as Pearson correlation coefficient (p value).

Keywords: Assessment; Robotic surgery; Surgical simulator; Training.