Limb loading enhances skill transfer between augmented and physical reality tasks during limb loss rehabilitation

Abstract

Background: Virtual and augmented reality (AR) have become popular modalities for training myoelectric prosthesis control with upper-limb amputees. While some systems have shown moderate success, it is unclear how well the complex motor skills learned in an AR simulation transfer to completing the same tasks in physical reality. Limb loading is a possible dimension of motor skill execution that is absent in current AR solutions that may help to increase skill transfer between the virtual and physical domains.

Methods: We implemented an immersive AR environment where individuals could operate a myoelectric virtual prosthesis to accomplish a variety of object relocation manipulations. Intact limb participants were separated into three groups, the load control (CG_LD; [Formula: see text]), the AR control (CG_AR; [Formula: see text]), and the experimental group (EG; [Formula: see text]). Both the CG_AR and EG completed a 5-session prosthesis training protocol in AR while the CG_LD performed simple muscle training. The EG attempted manipulations in AR while undergoing limb loading. The CG_AR attempted the same manipulations without loading. All participants performed the same manipulations in physical reality while operating a real prosthesis pre- and post-training. The main outcome measure was the change in the number of manipulations completed during the physical reality assessments (i.e. completion rate). Secondary outcomes included movement kinematics and visuomotor behavior.

Results: The EG experienced a greater increase in completion rate post-training than both the CG_AR and CG_LD. This performance increase was accompanied by a shorter motor learning phase, the EG's performance saturating in less sessions of AR training than the CG_AR.

Conclusion: The results demonstrated that limb loading plays an important role in transferring complex motor skills learned in virtual spaces to their physical reality analogs. While participants who did not receive limb loading were able to receive some functional benefit from AR training, participants who received the loading experienced a greater positive change in motor performance with their performance saturating in fewer training sessions.

Keywords: Augmented reality; Myoelectric control; Proprioception; Upper-limb prostheses.

Fig. 1

An overview of the augmented reality (AR) system. The head-mounted display (HMD) uses anterior, stereo cameras to pass-through the physical environment to the participant. This pass-through video is combined with virtual representations of a prosthetic limb as well as the AR-PHAM to create the AR scene. Virtual objects are anchored to locations in the physical environment through the kinematic trackers. Participants are tasked to complete a series of object manipulations in this AR environment. To control the virtual prosthesis, participants wear an EMG band on their dominant limb. The resultant EMG signals are then fed into an LDA classifier, translating the EMG data into one of five hand movement classes: rest (RE), hand open (HO), hand close (HC), wrist pronate (WP), or wrist supinate (WS)

Fig. 2

The experimental setup for the physical assessment. Participants use their EMG signals to control a multi-articulated prosthetic hand and wrist attached via a bypass socket. Participants also wear multiple trackers that measure their limb kinematics

Fig. 3

An illustration of the 4 rehabilitation tasks for the physical and augmented reality PHAM. Cylinders will start in an initial location and orientation (red) with the corresponding target location and orientation being highlighted on the PHAM frame (green). Each rehabilitation task requires: a positional change in elevation, a positional crossing of the participant’s midline, and a 90° change in object orientation

Fig. 4

Different stages of a user completing an AR-PHAM rehabilitation task. a The AR-PHAM is anchored in the user’s physical environment and prompts the user to begin a task by pressing the central red button. b Once started, the task spawns a red cylinder on the AR-PHAM frame and prompts a target location and orientation by highlighting an AR-PHAM object holder red. c The user then reaches for and manipulates the cylinder using the virtual prosthesis, controlling the device via their EMG signals and the LDA classifier. d The user transports the cylinder to the target location and releases it onto the object holder with the appropriate orientation

Fig. 5

The four phases of a successfully completed AR-PHAM rehabilitation task. a Beginning with the Reach phase, the user visually focuses the red cylinder and moves the virtual prosthesis into the cylinder’s proximity. b Followed by the Grasp phase, the user operates the virtual prosthesis via myoelectric control to grab the cylinder. c During the Transport phase, the user visually focuses the cylinder’s destination and moves the virtual prosthesis to relocate the cylinder. d In the final Release phase, the user again operates the virtual prosthesis to place the correctly aligned cylinder onto the destination location

Fig. 6

Maximum probability of a classification error ($P_e$) in the initial training set for each session for all IL participants using the LDA algorithm. For all IL participants, $P_e$ did not exceed 4%. Sample size, $N=8$ for days 2 through 6, $N=12$ otherwise

Fig. 7

Change in performance between the pre- and post-test PHAM. Participants in the experimental group (EG) experienced an average increase in task completion rate of 31.25%, while participants in the load (CG_LD) and AR (CG_AR) control groups experienced an average increase in performance of 6.25% and 6.88%, respectively. Error bars denote standard deviation. *Denotes $p<0.05$ for the Mann–Whitney Test. Sample size, $N=4$

Fig. 8

Performance of participants during the AR-PHAM training. Participants in the CG_AR experienced an average increase in performance of + 31.25% during training while participants in the EG experienced a + 16.88% increase over the same time period. Despite a greater change in performance, the EG still outperformed the CG_AR in the final training session: 59.38% vs 47.5%. Error bars denote standard deviation. *Denotes $p<0.05$ for the Mann–Whitney Test. Sample size, $N=4$

Fig. 9

Upper-limb joint range of motion (RoM) during successfully completed tasks. (Top) In both the pre-test and post-test, all IL participants presented similar joint RoM on average for all upper-limb DoF. (Bottom) Across AR training all sessions, the EG presented a greater mean RoM than the CG_AR in the following upper-limb DoF: shoulder flexion ($82.34\pm 33.47$° vs $49.79\pm 21.36$°), shoulder abduction ($44.38\pm 20.14$° vs $22.76\pm 15.02$°), and elbow flexion ($80.80\pm 29.90$° vs $46.62\pm 22.21$°). Error bars denote standard deviation. *Denotes $p<0.05$ for the Mann–Whitney Test. Sample size, $N=4$

Fig. 10

Percentage of time spent visually fixating to the prosthetic hand during object transport during successfully completed tasks. Across all sessions, the EG presented a greater mean fixation percentage ($90.37\pm 13.32\%$) than the CG_AR ($64.09\pm 23.85\%$). Error bars denote standard deviation. *Denotes $p<0.05$ for the Mann–Whitney Test. Sample size, $N=4$