Robotic hand augmentation drives changes in neural body representation

Abstract

Humans have long been fascinated by the opportunities afforded through augmentation. This vision not only depends on technological innovations but also critically relies on our brain's ability to learn, adapt, and interface with augmentation devices. Here, we investigated whether successful motor augmentation with an extra robotic thumb can be achieved and what its implications are on the neural representation and function of the biological hand. Able-bodied participants were trained to use an extra robotic thumb (called the Third Thumb) over 5 days, including both lab-based and unstructured daily use. We challenged participants to complete normally bimanual tasks using only the augmented hand and examined their ability to develop hand-robot interactions. Participants were tested on a variety of behavioral and brain imaging tests, designed to interrogate the augmented hand's representation before and after the training. Training improved Third Thumb motor control, dexterity, and hand-robot coordination, even when cognitive load was increased or when vision was occluded. It also resulted in increased sense of embodiment over the Third Thumb. Consequently, augmentation influenced key aspects of hand representation and motor control. Third Thumb usage weakened natural kinematic synergies of the biological hand. Furthermore, brain decoding revealed a mild collapse of the augmented hand's motor representation after training, even while the Third Thumb was not worn. Together, our findings demonstrate that motor augmentation can be readily achieved, with potential for flexible use, reduced cognitive reliance, and increased sense of embodiment. Yet, augmentation may incur changes to the biological hand representation. Such neurocognitive consequences are crucial for successful implementation of future augmentation technologies.

Fig. 1. Experimental design.

(A-C) The Third Thumb is a 3D-printed robotic thumb. Mounted on the side of the palm (1), the Thumb is actuated by two motors (fixed to a wrist band), allowing for independent control over flexion/adduction. The Thumb is powered by (2) an external battery, strapped around the arm and wirelessly controlled by (3) two force sensors fixed to the underside of the participant’s big toes. (D) Experimental design for the augmentation group. (E) Examples of the in-lab training tasks used for hand-Thumb collaboration, shared and Thumb individuation. Augmentation participants showed significant performance improvements on all of the tasks across training session. Asterisks denote significant effect of time at *** p<0.001. See Fig S2. For statistical quantification of the improvements seen in the control group.

Fig. 2. Behavioural correlates of hand augmentation.

(A-C) Augmentation participants showed significant daily improvement on the hand-Thumb coordination task. (D) Motor performance with the Thumb was not impacted by increased cognitive load during the first and last training days. (E) Augmentation participants showed greater improvement than controls on a hand-Thumb coordination task conducted before and after the training period. Participants showed improved performance even while blindfolded, indicating increased Thumb proprioception. (F) Self-reported Thumb embodiment increased significantly in the augmentation group following Thumb training. (G) Hand kinematics data collected during the training sessions. The first principal component (synchronised movement across all five fingers) captured less variance in the augmentation group compared to controls, indicating less synchronised movements. (H-I) The augmentation group showed lower inter-finger coupling, relative to controls during Thumb use, indicating change to the natural finger coordination. The bars depict group means, error bars represent standard error of the mean. Individual dots correspond to individual subjects’ average inter-finger (D1-D5) coordination scores as predicted by the linear mixed model (see Materials and Methods). Asterisks denote significant effects at * p<0.05, ** p<0.01 and *** p<0.001.

Fig. 3. Biological hand’s representation shrinks following hand augmentation.

(A) The sensorimotor hand area was defined anatomically, based on a primary motor cortex segmentation. (B) Group mean dissimilarity matrix of the right (augmented) hand pre- and post- training. Each cell shows the Mahalanobis (cross-validated) distance between the representational pattern of two fingers. (C) The average inter-finger distances of the right (augmented), but not the left (non-augmented) hand decreased significantly following Thumb use. The bars depict group mean, error bars represent standard error of the mean. Individual dots correspond to individual participants’ average distance as predicted by the linear mixed model (see Materials and Methods). (D) Multidimensional scaling (MDS) depiction of the left and right (augmented) hand representational structures. Ellipses indicate between-participant standard errors. Darker colours represent the post scan, whereas lighter colours represent the pre (baseline) scan. Red = D1, Yellow = D2, Green = D3, Blue = D4, Purple = D5. (E) SMA ROI was defined anatomically, based on BA6 segmentation (F) The distance between the hand and the feet, quantified in SMA, decreases significantly for the right, but not the left hand. (G) MDS depiction of the inter-body-part distances in the SMA. Darker colours represent the post scan, whereas lighter colours represent the pre (baseline) scan. Blue = Toes, Orange = Hand, Black = Lips. Asterisks denote significant effects at * p<0.05, ** p<0.01 and *** p<0.001.

Fig. 4. Outcomes of the computational simulation.

The adoption of an extra robotic thumb by an adult with a stable hand representation promotes a change to existing brain organisation. Here, we used computational simulations to explore multiple potential plasticity mechanisms that could trigger the observed shrinkage of the hand representation. First, based on information processing theories (47, 48), the integration of a new additional finger into the hand’s motor control could impinge on the existing representation of the biological fingers (new digit model). Second, the change in finger coordination, observed during training, may also lead to abrupt changes in excitability profiles that can trigger homeostatic plasticity mechanisms and promote increased tonic inhibition (homeostatic inhibition model (49)). Thirdly, the change to finger coordination may also result in increased finger individuation, leading to increased cortical representation of individual fingers via Hebbian learning (cortical magnification model, (50)). Simulating “neuronal activity” over a fixed-size ROI split into finger specific areas, we found that each of these processes is conceptually capable of causing the observed reduction in representational selectivity. (A) Mean dissimilarity matrices computed from 10000 simulations of each of the models. (B) Average distance (dissimilarity) is significantly decreased, as compared to the canonical hand representation in each of the models. Solid lines represent the mean of 10000 simulations, dashed lines denote the 1^st and 3^rd quartile of the data. Asterisks denote significant effects at *** p<0.001.