Objective assessment in residency-based training for transoral robotic surgery

Abstract

Objectives/hypothesis: To develop a robotic surgery training regimen integrating objective skill assessment for otolaryngology and head and neck surgery trainees consisting of training modules of increasing complexity leading up to procedure-specific training. In particular, we investigated applications of such a training approach for surgical extirpation of oropharyngeal tumors via a transoral approach using the da Vinci robotic system.

Study design: Prospective blinded data collection and objective evaluation (Objective Structured Assessment of Technical Skills [OSATS]) of three distinct phases using the da Vinci robotic surgical system in an academic university medical engineering/computer science laboratory setting.

Methods: Between September 2010 and July 2011, eight otolaryngology-head and neck surgery residents and four staff experts from an academic hospital participated in three distinct phases of robotic surgery training involving 1) robotic platform operational skills, 2) set up of the patient side system, and 3) a complete ex vivo surgical extirpation of an oropharyngeal tumor located in the base of tongue. Trainees performed multiple (four) approximately equally spaced training sessions in each stage of the training. In addition to trainees, baseline performance data were obtained for the experts. Each surgical stage was documented with motion and event data captured from the application programming interfaces of the da Vinci system, as well as separate video cameras as appropriate. All data were assessed using automated skill measures of task efficiency and correlated with structured assessment (OSATS and similar Likert scale) from three experts to assess expert and trainee differences and compute automated and expert assessed learning curves.

Results: Our data show that such training results in an improved didactic robotic knowledge base and improved clinical efficiency with respect to the set up and console manipulation. Experts (e.g., average OSATS, 25; standard deviation [SD], 3.1; module 1, suturing) and trainees (average OSATS, 15.9; SD, 3.9; week 1) are well separated at the beginning of the training, and the separation reduces significantly (expert average OSATS, 27.6; SD, 2.7; trainee average OSATS, 24.2; SD, 6.8; module 3) at the conclusion of the training. Learning curves in each of the three stages show diminishing differences between the experts and trainees, which is also consistent with expert assessment. Subjective assessment by experts verified the clinical utility of the module 3 surgical environment, and a survey of trainees consistently rated the curriculum as very useful in progression to human operating room assistance.

Conclusions: Structured curricular robotic surgery training with objective assessment promises to reduce the overhead for mentors, allow detailed assessment of human-machine interface skills, and create customized training models for individualized training. This preliminary study verifies the utility of such training in improving human-machine operations skills (module 1), and operating room and surgical skills (modules 2 and 3). In contrast to current coarse measures of total operating time and subjective assessment of error for short mass training sessions, these methods may allow individual tasks to be removed from the trainee regimen when skill levels are within the standard deviation of the experts for these tasks, which can greatly enhance overall efficiency of the training regimen and allow time for additional and more complex training to be incorporated in the same time frame.

Figure 1

Data collection setup. The Johns Hopkins University (JHU) Visual Imaging and Surgical Robotics Laboratory archival station acquires both endoscopic video and hand motion without the need for subject intervention. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.] © IF THIS IMAGE HAS BEEN PROVIDED BY OR IS OWNED BY A THIRD PARTY, AS INDICATED IN THE CAPTION LINE, THEN FURTHER PERMISSION MAY BE NEEDED BEFORE ANY FURTHER USE. PLEASE CONTACT WILEY’S PERMISSIONS DEPARTMENT ON PERMISSIONS@WILEY.COM OR USE THE RIGHTSLINK SERVICE BY CLICKING ON THE ‘REQUEST PERMISSION’ LINK ACCOMPANYING THIS ARTICLE. WILEY OR AUTHOR OWNED IMAGES MAY BE USED FOR NON-COMMERCIAL PURPOSES, SUBJECT TO PROPER CITATION OF THE ARTICLE, AUTHOR, AND PUBLISHER.

Figure 2

Study training tasks. Each subject practiced four training tasks in module 1, three (cart setup, port placement, workspace test) in module 2, and an ex vivo procedure in module 3. (A) Module 1. Four tasks for first module (left), endoscopic workspace test for module 2 (right). (B) Module 2 patient cart, port, and instrument setup. (C) Tumor excision and margin measurement in module 3. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.] © IF THIS IMAGE HAS BEEN PROVIDED BY OR IS OWNED BY A THIRD PARTY, AS INDICATED IN THE CAPTION LINE, THEN FURTHER PERMISSION MAY BE NEEDED BEFORE ANY FURTHER USE. PLEASE CONTACT WILEY’S PERMISSIONS DEPARTMENT ON PERMISSIONS@WILEY.COM OR USE THE RIGHTSLINK SERVICE BY CLICKING ON THE ’REQUEST PERMISSION’ LINK ACCOMPANYING THIS ARTICLE. WILEY OR AUTHOR OWNED IMAGES MAY BE USED FOR NON-COMMERCIAL PURPOSES, SUBJECT TO PROPER CITATION OF THE ARTICLE, AUTHOR, AND PUBLISHER.

Figure 3

Selected learning curves. OSATS scores range from 0 to 30 over the four sessions and three modules. Experts appear in the upper left hand corner. In future iterations of the training regimen, the average expert scores will be used to graduate trainees to the next module. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.] © IF THIS IMAGE HAS BEEN PROVIDED BY OR IS OWNED BY A THIRD PARTY, AS INDICATED IN THE CAPTION LINE, THEN FURTHER PERMISSION MAY BE NEEDED BEFORE ANY FURTHER USE. PLEASE CONTACT WILEY’S PERMISSIONS DEPARTMENT ON PERMISSIONS@WILEY.COM OR USE THE RIGHTSLINK SERVICE BY CLICKING ON THE ’REQUEST PERMISSION’ LINK ACCOMPANYING THIS ARTICLE. WILEY OR AUTHOR OWNED IMAGES MAY BE USED FOR NON-COMMERCIAL PURPOSES, SUBJECT TO PROPER CITATION OF THE ARTICLE, AUTHOR, AND PUBLISHER.

Figure 4

Task completion times and margins over four sessions of respective modules. Experts appear at the bottom left. Task completion times decrease and assessed task quality improves with training (compared with Fig. 3). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.] © IF THIS IMAGE HAS BEEN PROVIDED BY OR IS OWNED BY A THIRD PARTY, AS INDICATED IN THE CAPTION LINE, THEN FURTHER PERMISSION MAY BE NEEDED BEFORE ANY FURTHER USE. PLEASE CONTACT WILEY’S PERMISSIONS DEPARTMENT ON PERMISSIONS@WILEY.COM OR USE THE RIGHTSLINK SERVICE BY CLICKING ON THE ’REQUEST PERMISSION’ LINK ACCOMPANYING THIS ARTICLE. WILEY OR AUTHOR OWNED IMAGES MAY BE USED FOR NON-COMMERCIAL PURPOSES, SUBJECT TO PROPER CITATION OF THE ARTICLE, AUTHOR, AND PUBLISHER.