From monkeys to humans: observation-basedEMGbrain-computer interface decoders for humans with paralysis

Abstract

Objective. Intracortical brain-computer interfaces (iBCIs) aim to enable individuals with paralysis to control the movement of virtual limbs and robotic arms. Because patients' paralysis prevents training a direct neural activity to limb movement decoder, most iBCIs rely on 'observation-based' decoding in which the patient watches a moving cursor while mentally envisioning making the movement. However, this reliance on observed target motion for decoder development precludes its application to the prediction of unobservable motor output like muscle activity. Here, we ask whether recordings of muscle activity from a surrogate individual performing the same movement as the iBCI patient can be used as target for an iBCI decoder.Approach. We test two possible approaches, each using data from a human iBCI user and a monkey, both performing similar motor actions. In one approach, we trained a decoder to predict the electromyographic (EMG) activity of a monkey from neural signals recorded from a human. We then contrast this to a second approach, based on the hypothesis that the low-dimensional 'latent' neural representations of motor behavior, known to be preserved across time for a given behavior, might also be preserved across individuals. We 'transferred' an EMG decoder trained solely on monkey data to the human iBCI user after using Canonical Correlation Analysis to align the human latent signals to those of the monkey.Main results. We found that both direct and transfer decoding approaches allowed accurate EMG predictions between two monkeys and from a monkey to a human.Significance. Our findings suggest that these latent representations of behavior are consistent across animals and even primate species. These methods are an important initial step in the development of iBCI decoders that generate EMG predictions that could serve as signals for a biomimetic decoder controlling motion and impedance of a prosthetic arm, or even muscle force directly through functional electrical stimulation.

Keywords: EMG decoding; brain computer interface; monkey; paralyzed human; transfer learning.

Figure 1.

Cross-user decoding of EMG. (a), We recorded neural firing rates from M1 and EMGs from forearm and wrist muscles of a ‘source’ monkey trained to perform the isometric wrist task. We also recorded M1 data from a ‘target’ monkey. (b), The firing rates of the source monkey were projected in a low-dimensional neural manifold and the resulting latent signals used as input to train a source-monkey iBCI decoder. When using the target-monkey M1 data, this decoder fails to accurately predict EMGs. (c), In a first approach, we achieved source monkey EMG predictions by training a decoder between the target monkey M1 signals and the source monkey EMG signals. We call this approach ‘direct decoding’ (d), In a second approach, we computed a decoder solely from the source monkey data, and preprocessed the target M1 latent signals to align them to those of the source monkey. We used the aligned latent signals as input to the source-monkey decoder to obtain predicted EMG. We call this approach ‘transfer decoding’.

Figure 2.

Direct decoding of EMG across monkeys. (a), Cross-monkey EMG predictions obtained with target-monkey latent signals via direct decoder (blue lines). Data for a representative pair of monkeys for two trials in each target direction (directions separated by vertical dashed lines and indicated by arrows at the bottom of each column). These predictions are almost as good as those obtained by using the source-monkey latent signals as input to the decoder (grey lines). The $R^2$ values for both within- (grey) and cross- (blue) monkey decoding are shown for each muscle, computed relative to actual EMG recordings of the source monkey (black lines). (b), Overall cross-monkey decoding accuracy ($R^2$) with direct decoding for all pairs of monkeys. A kernel density estimate plot at the top shows the distribution of cross-monkey (off-diagonal) $R^2$ (blue) compared to within-monkey (on-diagonal) $R^2$ (grey).

Figure 3.

Latent neural signals become more similar across monkeys after neural alignment. (a), Representative latent signals described by the first two principal components for a source (top left) and target (bottom left) monkeys. We used CCA to transform the latent signals such that they were maximally correlated (center). The CCA-transformed signals are more similar, but neither looks like the original source-monkey trajectories. Consequently, we further transformed the target-monkey latent signals in the CCA aligned space by the using the inverse of the source-monkey CCA transformation ($C_S^{-1}$) to map them back into the original source-monkey latent coordinates (bottom right). Data were averaged across all trials for each target direction; single trial trajectories are shown as lighter traces. Arrows indicate the temporal evolution of the trajectories (from 0.5 s before to 1 s after cursor movement onset). (b), Overall cross-monkey latent signals similarity ($R^2$) after CCA alignment for all pairs of monkeys. A kernel density estimate plot shows the distribution of cross-monkey latent signals $R^2$ similarity after CCA alignment.

Figure 4.

Transfer decoding of EMG across monkeys. Overall cross-monkey decoding accuracy ($R^2$) with transfer decoding for all pairs of monkeys. A kernel density estimate plot shows the distribution of cross-monkey $R^2$ (red) compared to within-monkey $R^2$ (grey) as in figure 2

Figure 5.

Comparing direct decoding and transfer decoding. (a), Representative cross-monkey EMG predictions obtained with direct (blue lines) and transfer decoding (red lines) compared to source monkey ground truth (black lines, as in figure 2(a)). (b), Element-by-element scatter plot comparing direct and transfer decoding performance. Each point represents the within-monkey/cross-session (purple) or cross-monkey (green) accuracy of transfer decoding (x-axis) versus the relative accuracy of direct decoding (y-axis) of a single muscle. Direct decoding generally yielded higher decoding accuracy for both cross-monkey and cross-session predictions, especially when decoding accuracy was low. (c), Element-by-element scatter plot for the matrices in figures 4 and 2(b). Latent signals of monkey pairs with higher similarity after CCA alignment yielded higher decoding accuracy for both cross-monkey (circle dots) and within-monkey/cross-session (x signs) predictions.

Figure 6.

Accuracy of monkey-to-human EMG decoding depends on the latency relative to the go cue used for human trial segmentation. (a), Single-trial M1 data from the source monkey (top, first session from monkey J) and the human participant (bottom) projected onto the first principal component (computed in each case using the entire corresponding dataset) for a pair of oppositely directed targets (left: yellow, right: purple). The neural responses recorded during the human’s attempted task had greater trial-by-trial variation in timing and magnitude compared to those of the monkey. (b), EMG decoding accuracy with direct (red) and transfer decoding (blue) as a function of the time index relative to the go cue used to segment the human trials. The vertical dashed line indicates the time at which greatest EMG decoding accuracy is achieved; note that it precedes the go cue.

Figure 7.

EMG decoding from a human with tetraplegia. (a), EMG predictions for the four major wrist muscles obtained via direct (blue lines) and transfer decoding (red lines). Actual EMG recordings of the source monkey (black) provide a ground truth. (b), Latent M1 trajectories described by the first two principal components of the M1 data (left) recorded as the participant attempted to perform a wrist isometric task, and those for the source monkey (center; first session from monkey J). The human neural recordings had a stereotypical low-dimensional structure for each of the eight target directions, with large inter-trial variability (shown by the lighter traces). Despite this increased variability, CCA alignment recovered a shape with better similarity to that of the source monkey’s latent signals (right).

Figure 8.

A fixed direct decoder needs to be aligned across time. (a), EMG prediction accuracy over time using a fixed ‘day-0’ direct decoder trained on the first target monkey session. We compared the performance of the fixed direct decoder before (turquoise) and after (blue) within-monkey/across-time CCA alignment between the day-k and day-0 target neural data. CCA alignment stabilized the performance of the fixed direct decoder over time, such that it achieved performance similar to that of the transfer decoding approach (red).

Figure 9.

Task generalization of cross-monkey EMG decoding. Generalizability of the direct (blue) and transfer (red) decoding was assessed by training either the direct decoder or the transfer decoder using only a subset of all eight movement directions. (a), Task generalization when interpolating (i.e. training on cardinal directions and testing on diagonal directions); (b), Task generalization when extrapolating (i.e. training on adjacent upper directions and testing on lower directions). (c), Violin plots for the overall cross-monkey decoding accuracy for all pairs of monkeys when training and testing on all target directions (center), when interpolating (left), and when extrapolating (right). When interpolating, cross-monkey decoding is still possible with both direct and transfer decoding, albeit with lower accuracy. When extrapolating, the cross-monkey decoding generally failed, as indicated by the negative $R^2$ values.