A learning-based approach to artificial sensory feedback leads to optimal integration

Abstract

Proprioception-the sense of the body's position in space-is important to natural movement planning and execution and will likewise be necessary for successful motor prostheses and brain-machine interfaces (BMIs). Here we demonstrate that monkeys were able to learn to use an initially unfamiliar multichannel intracortical microstimulation signal, which provided continuous information about hand position relative to an unseen target, to complete accurate reaches. Furthermore, monkeys combined this artificial signal with vision to form an optimal, minimum-variance estimate of relative hand position. These results demonstrate that a learning-based approach can be used to provide a rich artificial sensory feedback signal, suggesting a new strategy for restoring proprioception to patients using BMIs, as well as a powerful new tool for studying the adaptive mechanisms of sensory integration.

Figure 1

Behavioral task and sensory feedback. (a) Timeline of a behavioral trial (see Online Methods for details). (b) Visual feedback of the instantaneous movement vector (black arrow) takes the form of a random moving–dot flow–field (“dot–field”). The coherence of the dot–field—the percentage of dots moving in the same direction—determines its reliability. (c) Implantation site of stimulating electrode arrays for monkeys D (black) and F (blue). CS–central sulcus; IPS–inferior parietal sulcus. Right: the assigned PD of each stimulating electrode is overlaid on its location within the array. (d) An example ICMS trial showing the movement vector at the beginning of the reach (black arrow) and the monkey’s subsequent movement path (blue). At right: ICMS patterns delivered during the trial; each row represents the time–varying stimulation pattern of the electrode with the preferred direction (PD) indicated at left (black arrow). Vermillion tick marks denote biphasic stimulation pulses, which are shown subsampled for clarity. (e) Inset: the instantaneous movement vectors encoded at two time–points during the reach are shown as solid and dashed black arrows. Below, the pattern of stimulation encoding each movement vector is shown across electrodes; arrowheads indicate the PD of each electrode.

Figure 2

Comparison of task performance across sensory feedback conditions. Behavioral performance measures are averaged across the last seven testing sessions for each monkey, shown for each sensory feedback type and as a function of visual coherence (for VIS and VIS+ICMS trials). Error bars denote bootstrapped standard error of the mean. The ICMS data points, which are independent of visual coherences, are extended across the plot to aid visual comparison. (a) number of movement sub–segments; (b) movement path length, normalized by the initial distance to the reach target; (c) movement time, normalized as in b. See online methods for a detailed description of task performance measures.

Figure 3

Evolution of performance over training (Monkey F). Behavioral performance measures are shown as a function of the cumulative number of VIS+ICMS trials performed (training and testing). The data, collected during testing sessions, were smoothed for clarity (Gaussian window with standard deviation of 2.8 training sessions, translating to approximately 2,800 training trials for Monkey F). The visual coherence on training trials was decreased across training sessions (indicated by gray bars at the bottom of the figure and vertical gray lines at the transitions). The left, thin green line denotes the onset of ICMS–only trials, where target sizes were temporarily larger than in the other trial conditions; the right, thick green line denotes the beginning of ICMS–trials with targets of standard size. (a) percent correct trials; (b) number of movement segments measured online error corrections; (c) movement time for the trial is normalized by the initial distance to the reach target; (d) path length, normalized as in c. See Supplementary Table 2 for additional details on the training and testing schedule.

Figure 4

Monkeys estimate both target distance and direction from sensory feedback. Vermillion points reflect performance with ICMS and purple points reflect feedback with VIS. Black solid lines are unity and the thick colored lines represent linear fits between the movement and target variable. Fits were performed separately for the distal (target angle [0:π]) and proximal (target angle [−π:0]) halves of the workspace. (a,b) Initial movement angle versus target angle for monkey D and F, respectively. (c,d) Initial movement distance versus target distance for monkey D and F, respectively. Region within the dashed black lines falls within the diameter of the target.

Figure 5

Integration of vision and ICMS minimizes reach variance. (a) Standard deviation of initial angle relative to target angle as a function of visual coherence for different feedback conditions for each monkey. Standard deviation was calculated after subtracting a smoothed estimate of mean initial angle (panel b); results were qualitative unchanged with only the target angle subtracted (i.e., angle computed with respect to straight–line reach; Supplementary Fig. 5). Error bars represent standard error of the mean. Dashed black lines indicate model predictions with no motor noise (Online Methods, Eqn. 3a); dotted black lines indicate model predictions with maximal motor noise (Online Methods, Eqn. 3b). (b) Mean initial angle, with respect to a straight–line reach. Smoothed values are shown on a polar plot as a function of target direction. Data is from Monkey D; for Monkey F, see Supplementary Fig. 5b. (c) Visual cue weighting (see Online Methods) for combined VIS+ICMS conditions were closer to zero (ICMS) for low coherence trials and closer to one (VIS) for high coherence trials. Blue filled circles: visual cue weighting estimated from data (Online Methods, Eqn. 6); black unfilled circles: minimum variance model prediction (Online Methods, Eqn. 5). Data is from Monkey D; for Monkey F, see Supplementary Figure 5c.

Figure 6

Directed error correction. (a) Angle of the second sub–movement as a function of instantaneous movement vector angle in trials that required error correction. Top: ICMS–only trials; Bottom: VIS–only trials at high visual coherence (100% for Monkey D, 50% for Monkey F). Black line: unity. (b) Error variance (rad²) in sub–movement angle estimation for ICMS and VIS. Dashed vermillion line denotes chance (random, undirected movement). Error bars represent standard error of the mean.