An Accurate and Rapidly Calibrating Speech Neuroprosthesis

Abstract

Background: Brain-computer interfaces can enable communication for people with paralysis by transforming cortical activity associated with attempted speech into text on a computer screen. Communication with brain-computer interfaces has been restricted by extensive training requirements and limited accuracy.

Methods: A 45-year-old man with amyotrophic lateral sclerosis (ALS) with tetraparesis and severe dysarthria underwent surgical implantation of four microelectrode arrays into his left ventral precentral gyrus 5 years after the onset of the illness; these arrays recorded neural activity from 256 intracortical electrodes. We report the results of decoding his cortical neural activity as he attempted to speak in both prompted and unstructured conversational contexts. Decoded words were displayed on a screen and then vocalized with the use of text-to-speech software designed to sound like his pre-ALS voice.

Results: On the first day of use (25 days after surgery), the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. Calibration of the neuroprosthesis required 30 minutes of cortical recordings while the participant attempted to speak, followed by subsequent processing. On the second day, after 1.4 additional hours of system training, the neuroprosthesis achieved 90.2% accuracy using a 125,000-word vocabulary. With further training data, the neuroprosthesis sustained 97.5% accuracy over a period of 8.4 months after surgical implantation, and the participant used it to communicate in self-paced conversations at a rate of approximately 32 words per minute for more than 248 cumulative hours.

Conclusions: In a person with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore conversational communication after brief training. (Funded by the Office of the Assistant Secretary of Defense for Health Affairs and others; BrainGate2 ClinicalTrials.gov number, NCT00912041.).

Figure 1.. Electrode locations and speech decoding setup.

a, Diagram of the brain-to-text speech neuroprosthesis. Cortical neural activity is measured using four 64-electrode arrays. Machine learning techniques decode the cortical neural activity into an English phoneme every 80 ms (see also Section S5). b, Approximate microelectrode array locations (gray squares) superimposed on a 3d reconstruction of the participant’s brain. Colored regions correspond to cortical areas aligned to the participant’s brain using the Human Connectome Project’s MRI protocol scans before implantation.

Figure 2.. Online speech decoding performance.

Phoneme error rates (top) and word error rates (bottom) are shown for each session for two vocabulary sizes (50 versus 125,000 words). The ‘hours’ row of the horizontal axis reports the cumulative hours of neural data used to train the speech decoder for that session. Aggregate error rates across all evaluation sentences are shown for each session (mean ± 95% confidence interval). Vertical dashed lines represent when decoder improvements were introduced. Fig. S20 shows phoneme and word error rates for individual blocks.

Figure 3.. Conversation Mode user interface.

Photograph of the participant and speech neuroprosthesis in Conversation Mode. The neuroprosthesis detected when he was trying to speak solely based on neural activity, and concluded either after 6 seconds of speech inactivity, or upon his optional activation of an on-screen button via eye tracking. After the decoded sentence was finalized, the participant selected on-screen confirmation buttons via eye tracking to indicate if the decoded sentence was correct.

Figure 4.. Use of the neuroprosthesis for self-initiated speech.

a, Cumulative hours that the participant used the speech neuroprosthesis to communicate during structured research sessions and personal use. For the sessions outlined in blue, Conversation Mode decoding accuracy was quantified in (b). b, Histogram evaluating speech decoding accuracy in conversations for the n = 925 sentences with known true labels (Section S1.09). The average word error rate was 3.7% (95% CI, 3.3% to 4.3%). c, Self-reported decoding accuracy for each sentence across all Conversation Mode data (n = 21,829).