Enhancing Mirror Therapy via Scaling and Shared Control: A Novel Open-Source Virtual Reality Platform for Stroke Rehabilitation

Abstract

Mirror therapy is increasingly used in stroke rehabilitation to improve functional movements of the affected limb. However, the extent of mirroring in conventional mirror therapy is typically fixed (1:1) and cannot be tailored based on the patient's impairment level. Further, the movements of the affected limb are not actively incorporated in the therapeutic process. To address these issues, we developed an immersive VR system using HTC Vive and Leap Motion, which communicates with our free and open-source software environment programmed using SteamVR and the Unity 3D gaming engine. The mirror therapy VR environment was incorporated with two novel features: (1) scalable mirroring and (2) shared control. In the scalable mirroring, mirror movements were programmed to be scalable between 0 and 1, where 0 represents no movements, 0.5 represents 50% mirroring, and 1 represents 100% mirroring. In shared control, the contribution of the mirroring limb to the movements was programmed to be scalable between 0 to 1, where 0 represents 100% contribution from the mirroring limb (i.e., no mirroring), 0.5 represents 50% of movements from the mirrored limb and 50% of movements from the mirroring limb, and 1 represents full mirroring (i.e., no shared movements). Validation experiments showed that these features worked appropriately. The proposed VR-based mirror therapy is the first fully developed system that is freely available to the rehabilitation science community. The scalable and shared control features can diversify mirror therapy and potentially augment the outcomes of rehabilitation, although this needs to be verified through future experiments.

Keywords: illusion; low cost; mirror neurons; motor control; telehealth; virtual rehabilitation.

Fig. 1

A schematic of the (a) Sensors used and their recommended positions on the user. The headset located the position and orientation of the user’s head. The sensors on the user’s forearms and feet provided the kinematics for the arms and legs, respectively. A sensor fixed to the user’s back located the user’s trunk, and two SteamVR Base Stations (not pictured) mounted on opposing corners of the data collection area located the user in the virtual environment. A Leap Motion Controller provided kinematics for the user’s wrists, hands, and fingers; (b) Graphical user interface (GUI) of the virtual reality system. Through the GUI, the user could select different avatars to control, as well as alter the avatar’s anthropometry and skin tone; (c) Different avatars that could be controlled by the user in the virtual environment; and (d) Control to alter the skin tone of the avatar

Fig. 2

(a) Demonstration of stylus-based calibration. Calibrating the torso starts by touching the calibration stylus (HTC Vive Controller) to the back tracker, which assigns the tracker to the torso. The shoulder and hip joint centers are assumed to be fixed distances from the back tracker (torso is assumed to be rigid) and are found by touching opposite sides of each joint with the calibration stylus and taking the average. To calibrate the leg, the foot is assumed to be rigid, then the tracker is assigned to the foot. Following this, the ankle joint center is assumed to be a fixed distance from the foot tracker and is found by touching opposite sides of the joint with the stylus and taking the average. Then the orientation of the foot is found by aligning the stylus with the foot’s orientation. Like the leg, the arm is calibrated by assuming that the forearm is a rigid body and assigning the tracker to the correct limb and then finding the elbow joint center and forearm orientation. (b) A diagram detailing how the virtual reality system determines the proximal segments of the user’s extremities. The positions of the trackers are directly accessible to the program (green vectors). The shoulder, elbow, hip, and ankle joint centers are known from calibration (orange vector). The humerus is computed as the difference between the vectors to the elbow and shoulder joint centers. To determine the knee position, the program assumes the hip, knee, and ankle joint centers fall in the same plane, and uses closed-form inverse kinematics to determine the knee joint center from the vector from the hip to the ankle and the measured thigh and shank lengths

Fig. 3

A schematic of the different forms of mirror therapy. (a) Conventionally, mirror therapy creates a perfect reflection of the user’s limb. (b) In scalable mirroring, the virtual arm is no longer a perfect reflection of the user’s limb, instead exhibits motion that is scaled down from the user’s actual motion. (c) In shared control, the virtual limb is controlled by both limbs of the user

Fig. 4:

The end-point trajectories recorded by the virtual reality program while the user traced a fixed trajectory (a circle of 0.75m radius). Note that the end-points were accurately tracked (average error of about 5%) by the virtual reality system

Fig. 5

A schematic showing side-by-side comparisons of the user and the avatar in different postures in real-time. Note that the avatar tracked the user’s postures reasonably well without any noticeable differences

Fig. 6

Comparison of joint angles measured by the virtual reality program with joint angles measured using Tracker software

Fig. 7

Measured distances from a point on the pelvis in the user’s mid-sagittal plane to different joint centers on each arm during conventional mirroring. Note that the distances from the pelvis to the joint centers on each arm were nearly identical, indicating that the mirroring happened correctly across the user’s mid-sagittal plane

Fig. 8

Traces of the elbow flexion angles (0° is full elbow extension) during (a) scalable mirroring and (b) shared control. In (a) the user reached forward with their left arm only, keeping their right arm stationary. In (b) the user reached forward with their left arm, and then followed that reach with an identical reach using their right arm. Note that the joint angles relative to the reference pose and the joint angles relative to the user’s right arm scaled appropriately in both scalable mirroring and shared control, respectively. The dashed line (black) denotes the reference pose for scalable mirroring

Fig. 9

A schematic of bimanual control (a) and “freeze” or unimanual control (b) when a user was performing a grasping task in the virtual reality environment. Note, in the bimanual control, both the mirrored and the mirroring limb moved when performing the task, whereas, in the unimanual control, only the mirroring limb moved while the mirrored limb was frozen