Adaptive Surgical Robotic Training Using Real-Time Stylistic Behavior Feedback Through Haptic Cues

Abstract

Surgical skill directly affects surgical procedure outcomes; thus, effective training is needed to ensure satisfactory results. Many objective assessment metrics have been developed that provide the trainee with descriptive feedback about their performance however, often lack feedback on how to improve performance. The most effective training method is one that is intuitive, easy to understand, personalized to the user,and provided in a timely manner. We propose a framework to enable user-adaptive training using near real-time detection of performance, based on intuitive styles of surgical movements, and design a haptic feedback framework to assist with correcting styles of movement. We evaluate the ability of three types of force feedback (spring, damping, and spring plus damping feedback), computed based on prior user positions, to improve different stylistic behaviors of the user during kinematically constrained reaching movement tasks. The results indicate that five out of six styles studied here were improved using at least one of the three types of force feedback. Task performance metrics were compared in the presence of the three types of feedback. Task time was statistically significantly lower when applying spring feedback, compared to the other two types of feedback. Path straightness and targeting error were statistically significantly improved when using spring-damping feedback compared to the other two types of feedback. This study presents a groundwork for adaptive training in robotic surgery based on near real-time human-centric models of surgical behavior.

Keywords: Adaptive and Intelligent Educational Systems; Force Feedback; Surgical Robotics.

Figure 1:

System Block Diagram: The human user interacts with a haptic device and the simulation environment (a). Before the experiment, training movement data is used to learn a dictionary of stylistic features and a classifier is trained to predict stylistic deficiencies in near real-time [28] (b). During the experiment, kinematic measurements from the haptic device is represented into stylistic behaviors by projecting it on the learned dictionary (c). The quality of the user’s style is detected using a classifier which takes the coefficients of the new representation of the data as an input (d). Finally, force feedback is provided to the user if negative performance is detected. Three different types of force feedback were evaluated in this study for their effectiveness in improving user style. Feedback is computed from prior user positions and velocities (e).

Figure 2:

Three types of haptic feedback: spring, damping, and spring + damping feedback were studied here for their ability to provide stylistic cues to the human operator. A force feedback was generated based on the user’s prior position in time.

Figure 3:

(a) User interface: user interacting with simulated environment using the Geomagic Touch haptic device. The task was initiated by moving the virtual stylus to the red doughnut and would end by reaching the specified target. (b) Target layout.

Figure 4:

An example of an experiment protocol for one subject. The protocol consists of six blocks, each related to one stylistic behavior detection algorithm that was activated for that block. For each block, the user first performed a set of reaching movements with no feedback to enable a baseline computation of style, followed by a set of trials with feedback that was provided, based on measured stylistic deficiencies. For each subject, a single feedback method was provided throughout the whole experiment, but at different points of time, depending on the style detection algorithm for that subject. Hence, a unique feedback relevant to style was provided to each subject.

Figure 5:

Comparing the effects of three different types of haptic feedback on each style. For each group of subjects receiving the same type of feedback, the mean and standard deviation is shown for the number of positive performance normalized to the total number of detections and divided by the baseline stylistic positive performance, for each style. The values above 1 show an improvement in the performance when applying feedback compared to the no feedback condition.

Figure 6:

Target layout and resulting needle paths for all subjects. Green paths indicating smallest error.

Figure 7:

Three metrics including (a) time to complete the task, (b) target positioning error, (c) and needle trajectory straightness were used to evaluate the effect of the haptic cues on task performance. For each group (who received the same type of force feedback), the mean and standard deviation of each task performance metric is calculated and compared for 4 target locations.

Figure 8:

NASA-TLX