Decoding spoken words using local field potentials recorded from the cortical surface

Abstract

Pathological conditions such as amyotrophic lateral sclerosis or damage to the brainstem can leave patients severely paralyzed but fully aware, in a condition known as 'locked-in syndrome'. Communication in this state is often reduced to selecting individual letters or words by arduous residual movements. More intuitive and rapid communication may be restored by directly interfacing with language areas of the cerebral cortex. We used a grid of closely spaced, nonpenetrating micro-electrodes to record local field potentials (LFPs) from the surface of face motor cortex and Wernicke's area. From these LFPs we were successful in classifying a small set of words on a trial-by-trial basis at levels well above chance. We found that the pattern of electrodes with the highest accuracy changed for each word, which supports the idea that closely spaced micro-electrodes are capable of capturing neural signals from independent neural processing assemblies. These results further support using cortical surface potentials (electrocorticography) in brain-computer interfaces. These results also show that LFPs recorded from the cortical surface (micro-electrocorticography) of language areas can be used to classify speech-related cortical rhythms and potentially restore communication to locked-in patients.

Figure 1

The micro-ECoG grid and surgical placement. a, A single 16-electrode 4×4 micro-ECoG grid shown next to a U.S. quarter-dollar coin. b, Photograph of micro-ECoG surgical placement; the green wire bundle leads to the array over Wernicke’s area; the orange wire bundle leads to the array over face motor cortex. c, Electrode positions in situ; micro-ECoG grids in red and clinical ECoG electrodes in yellow.

Figure 2

Using frequency-domain structure to decode simultaneously from multiple channels. a, 500-msec windows temporally aligned to spoken words contain frequency-domain structure in a spectrogram of neural data recorded by a single micro-electrode over face motor cortex. Axis labels indicate that data from multiple micro-electrodes and trials will be used. b, Power spectra were calculated for each trial and each micro-electrode. c, For each trial, power spectra from all micro-electrodes were concatenated. Trials were stacked to form a large two-dimensional matrix of micro-electrode and trial information. d, Performing principal component analysis on this matrix generated a cluster for each word that allowed nearest-centroid classification.

Figure 3

Raw data, spectrogram, and mean power during conversation and task. a, Audio waveform of conversation and verbal task in which the patient repeated the word “yes.” b, Normalized spectrogram of neural data recorded from a single electrode over face motor cortex during the same time period shown in (a). c, Normalized spectrogram of neural data recorded from a single electrode over Wernicke’s area during the same time period shown in (a). d, Mean power and standard error between 70 and 200 Hz for the 16 electrodes over FMC and the 16 electrodes over Wernicke’s area.

Figure 4

Classification accuracy for combinations of two through ten words. The distribution of classification accuracies from performing each combination of two through ten words is shown. Results are shown using features from all 16 electrodes over FMC; features from all 16 electrodes over Wernicke’s area; and features from the best 5 electrodes over FMC.

Figure 5

Decodability matrices for FMC and Wernicke’s area. The classification accuracies of all word pairs are shown for FMC (left) and Wernicke’s area (right) using features from the best five electrodes over each area. In each square plot, the intersection of a row and column indicates the classifier’s accuracy for that word pair. The diagonal is marked with black squares indicating the irrelevant case of classifying a word against itself.

Figure 6

Topography of performance by individual electrode and word. a, The topography of classification accuracy is shown for the micro-electrodes resting over FMC. b, The topography of classification accuracy is shown for the micro-electrodes resting over Wernicke’s area.