Speech motor cortex enables BCI cursor control and click

Abstract

Objective.Decoding neural activity from ventral (speech) motor cortex is known to enable high-performance speech brain-computer interface (BCI) control. It was previously unknown whether this brain area could also enable computer control via neural cursor and click, as is typically associated with dorsal (arm and hand) motor cortex.Approach.We recruited a clinical trial participant with amyotrophic lateral sclerosis and implanted intracortical microelectrode arrays in ventral precentral gyrus (vPCG), which the participant used to operate a speech BCI in a prior study. We developed a cursor BCI driven by the participant's vPCG neural activity, and evaluated performance on a series of target selection tasks.Main results.The reported vPCG cursor BCI enabled rapidly-calibrating (40 s), accurate (2.90 bits per second) cursor control and click. The participant also used the BCI to control his own personal computer independently.Significance.These results suggest that placing electrodes in vPCG to optimize for speech decoding may also be a viable strategy for building a multi-modal BCI which enables both speech-based communication and computer control via cursor and click. (BrainGate2 ClinicalTrials.gov ID NCT00912041).

Keywords: brain-computer interface; cursor control; speech motor cortex; ventral precentral gyrus.

Figure 1.

Rapid calibration of cursor control during first-ever usage. (a) Schematic of the cursor BCI. Neural activity was recorded by four 64-electrode arrays located in the left ventral precentral gyrus. Neural features were calculated in bins of 10 ms. A linear velocity decoder predicted the participant’s intended velocity and moved a computer cursor accordingly. (b) 3-D reconstruction of T15’s brain (left view), with implanted Utah array locations (black squares) and brain areas (magenta, green, blue regions) overlaid. Brain areas were estimated through alignment of T15’s brain with the Human Connectome Project cortical atlas via preoperative MRI scans. (c) Trial-averaged firing rates (mean ± s.e.) recorded from three example electrodes during the Radial8 Calibration Task. Activity is aligned to when the target appeared (which was also the go cue), and colored by cued target direction (same colors as in panel e). Multiple arrays showed cursor-related modulation, and some electrodes showed directional tuning. Firing rates were Gaussian smoothed (std 50 ms) before trial-averaging. (d) Timeline of the first 3 min that T15 ever used a cursor BCI. Gray marks every few seconds (top) indicate updates to the cursor decoder. Angular error between the vector toward the target and the vector predicted by the decoder is plotted as a rolling average (20 s). The task began in open-loop (assist = 1.0) and gradually transitioned to closed-loop (assist = 0.0) as angular error lowered. Teal diamonds (bottom) indicate successful target acquisitions (reaching the target and dwelling for 0.8 s). Filled diamonds indicate fully closed-loop trials. (e) Center-out-and-back cursor trajectories (224 trials) during T15’s first six Radial8 Calibration Task blocks, colored by cued target direction.

Figure 2.

Accurate cursor control and click. (a) Trial-averaged firing rates (mean ± s.e.) recorded from four example electrodes during the Grid Evaluation Task. Activity is aligned to when the cursor entered the cued target (left), and then to when the click decoder registered a click (right). Firing rates were Gaussian smoothed (std 20 ms) before trial-averaging. (b) The Grid Evaluation Task. The participant attempted to move the cursor (white circle) to the cued target (green square) and click on it. (c) Location of every click that was performed, relative to the current trial’s cued target (central black square), during blocks with the 6 × 6 grid (left) and with the 14 × 14 grid (right). Small gray squares indicate where the cursor began each trial, relative to the trial’s cued target. (d) Timeline of the seventeen 3 min Grid Evaluation Task blocks. Each point represents a trial and indicates the trial length and trial result (success or failure). Each gray region is a single block. (e) T15’s online bitrate performance in the Grid Evaluation Task, compared to the highest-performing prior dPCG cursor control study. Circles are individual blocks (only shown for this study). Triangles are averages per participant (from this study and others).

Figure 3.

The dorsal 6v array contributed the most to cursor velocity decoding. (a)( Zoomed-in view of T15’s array locations shown in figure 1(b). Triangles indicate arrays providing the best decoding performance for speech (orange) and for cursor control (crimson). The best speech arrays were identified in Card et al. 2024. (b) Offline analysis of cursor decoders trained using neural features from each array alone (left) and with each array excluded (right), evaluated according to average angular error between the vector toward the target and the vector predicted by the decoder. The decoder trained on d6v alone performed almost the same as the decoder trained on all arrays. Error bars and the green shaded region indicate 95% CI. Stars indicate significant difference from chance performance (gray star, p < 0.01, one-sample t-test, Bonferroni correction) or from performance using all arrays (green star, p < 0.01, two-sample t-test, Bonferroni correction). (c) Offline classification of click vs. non-click time windows using neural features from each array alone (left) and with each array excluded (right), evaluated with classification accuracy. Each of the four array locations contained click information. Error bars and the green shaded region indicate 95% CI. Stars indicate significant difference from chance performance (gray star, p < 0.01, one-sample t-test, Bonferroni correction) or from performance using all arrays (green star, p < 0.01, two-sample t-test, Bonferroni correction). Per-electrode contribution maps for cursor and click are shown in supp. figure 3.

Figure 4.

Cursor control was impacted by simultaneously speaking. (a) Overview of the Simultaneous Speech and Cursor Task. Each trial may or may not have a speech go cue (an auditory beep), which could occur during the move period or dwell period. (b) Trial-averaged firing rates (mean ± s.e.) recorded from three example electrodes during the Simultaneous Speech and Cursor Task. Activity during cursor control alone (left and center columns) is aligned first to target presentation and then to cursor go cue, and is colored by cued target direction. Activity during simultaneous cursor control and attempted speech (right column) is aligned to speech go cue, and is colored by prompted word. Firing rates were Gaussian smoothed (std 50 ms) before trial-averaging. (c) Comparison of target acquisition times in different trial conditions in the Simultaneous Speech and Cursor Task. Target acquisition times in the only condition requiring speaking were significantly longer than the other conditions (* p < 0.05, ** p < 0.01, *** p < 0.001, rank-sum test, Bonferroni correction). Supp. figure 4(b) shows an offline version of this analysis using angular error instead of target acquisition time.

Figure 5.

Participant T15 controlled his personal desktop computer with the cursor BCI. a. Over-the-shoulder view of T15 neurally controlling the mouse cursor on his personal computer. The red arrow points to the cursor. (b) and (c). Screenshots of T15’s personal computer usage, with cursor trajectories (pink lines) overlaid. Cursor position every 250 ms (circles) and clicks (stars) are also drawn. In b., T15 first opened the settings application (left) and then switched his computer to light mode (right). In c., T15 opened Netflix from Chrome’s new tab menu (top) and then selected his Netflix user (bottom).