A click-based electrocorticographic brain-computer interface enables long-term high-performance switch-scan spelling

Abstract

Background: Brain-computer interfaces (BCIs) can restore communication in movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command "click" decoders provide a basic yet highly functional capability.

Methods: We sought to test the performance and long-term stability of click-decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis (ALS). We trained the participant's click decoder using a small amount of training data (< 44 minutes across four days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating.

Results: Using this click decoder to navigate a switch-scanning spelling interface, the study participant was able to maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation interrupted testing with this fixed model, a new click decoder achieved comparable performance despite being trained with even less data (< 15 min, within one day).

Conclusion: These results demonstrate that a click decoder can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.

Figure 1. Real-time decoding pipeline.

(a) The participant was seated upright with his forearms on the armrests of a chair facing a computer monitor where the switch-scanning speller application was displayed. (b) Position of both 64-electrode grids overlayed on the left cortical surface of the participant’s brain. The dorsal and ventral grids primarily covered cortical upper limb and face regions respectively. The electrodes are numbered in increasing order from left to right and from bottom to top. Magenta: pre-central gyrus; Orange: post-central gyrus. (c) ECoG voltage signals were streamed in 100 ms packets to update a 256 ms running buffer for real-time spectral pre-processing. A sample of signals from 20 channels is shown. (d) A Fast Fourier Transform filter was used to compute the spectral power of the 256 ms buffer, from which the HG log-power (110–170 Hz) was placed into a 1 s running buffer (10 feature vectors). (e) The running buffer was then used as time history for the recurrent neural network (RNN). (f) An RNN-FC (RNN-fully connected) network then predicted rest or grasp every 100 ms depending on the higher output probability. (g) Each classification result was stored as a vote in a 7-vote running buffer such that the number of grasp votes had to surpass a predetermined voting threshold (4-vote threshold shown) to initiate a click. A lock-out period of 1 s immediately followed every detected click to prohibit multiple clicks from occurring during the same attempted movement. (h) Once a click was detected, the switch scanning speller selected the highlighted row or element within that row. Two clicks were necessary to type a letter or autocomplete a word.

Figure 2. Long-term use of a fixed click detector.

(a) Training data was collected during 4 sessions that occurred within a period of 15 days. For each day, each sub-bar represents a separate block of training data collection (6 training blocks total). (b) Using the fixed decoder, one block of switch-scanning with the communication board was performed +21 days post-training data collection (purple). From Day +46 to Day +81, the fixed decoder was used for switch-scan spelling with a 7-vote threshold (blue). From Day +81 to Day +111, the fixed decoder was used for switch-scan spelling with a 4-vote threshold (teal). For each day, each sub-bar represents a separate spelling block of 3–4 sentences. The horizontal axis spanning both (a) and (b) represents the number of days relative to the last day of training data collection (Day 0).

Figure 3. Long-term switch-scanning spelling performance.

Across all subplots, triangular and circular markers represent metrics using a 7-vote and 4-vote voting threshold respectively. (a) Sensitivity of grasp detection for each session. Dashed line delineates 100% sensitivity. (b) True-positive and false-positive frequencies (TPF and FPF) measured as detections per minute. Dashed line delineates 0 FPF. (c) Average latencies with standard deviation error bars of grasp onset to algorithm detection and to on-screen click. The averages and standard deviations were computed from latency measurements across all spelling blocks from one session using the same voting threshold. Using 7-vote and 4-vote voting thresholds, onscreen clicks happened an average of 207 ms and 203 ms respectively after detection. Note that detection latencies were not registered in the first six sessions. (d) Correct characters and words per minute (CCPM and CWPM).

Figure 4. Channel importance for grasp classification.

Saliency maps for the model used in real-time, a model using HG features from all channels except from channel 112, and a model using HG features only from channels covering cortical hand-knob are shown in (a), (c) and (e) respectively. Electrodes overlayed with larger circles represent greater importance for grasp classification. White and transparent circles represent electrodes which were not used for model training. Mean confusion matrices from repeated 10-fold CV using models trained on HG features from all channels, all channels except for channel 112, and channels covering only the cortical hand-knob are shown in (b), (d), and (f) respectively. For all confusion matrices, the percent value in each element of the matrix represents how many times the validation features across all repetitions of all validation folds were predicted correctly or incorrectly. The mean classification accuracy was computed from averaging the values on the diagonal of the confusion matrix. (g) Box and whisker plot showing the offline classification accuracies from 10 cross-validated testing folds using models with the above-mentioned channel subsets. Specifically, for one model configuration, each dot represents the average accuracy of the same validation fold across 20 repetitions of 10-fold CV (see Methods: Channel contributions). Offline classification accuracies from CV-models trained on all features from all channels were statistically higher than CV-models trained on features from channels only over cortical hand-knob (*P = 0.015, Wilcoxon Rank-Sum test with 3-way Bonferroni-Holm correction). Offline classification accuracies from CV-models trained on features from all channels except for channel 112 were not statistically different from those trained on features from all channels or features only from channels only over cortical hand-knob.