A click-based electrocorticographic brain-computer interface enables long-term high-performance switch scan spelling

Abstract

Background: Brain-computer interfaces (BCIs) can restore communication for movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command click detectors provide a basic yet highly functional capability.

Methods: We sought to test the performance and long-term stability of click decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis. We trained the participant's click detector using a small amount of training data (<44 min across 4 days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating.

Results: Using a click detector to navigate a switch scanning speller interface, the study participant can maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation can interrupt usage of a fixed model, a new click detector can achieve comparable performance despite being trained with even less data (<15 min, within 1 day).

Conclusions: These results demonstrate that a click detector can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.

Fig. 1. Online click detection pipeline.

a The participant was seated upright with his forearms on the armrests of a chair facing a computer monitor where the switch scanning speller application was displayed. b Position of both 64-electrode grids overlayed on the left cortical surface of a virtual reconstruction of the participant’s brain. The dorsal and ventral grids primarily covered cortical upper limb and face regions, respectively. The electrodes are numbered in increasing order from left to right and from bottom to top. Magenta: pre-central gyrus; Orange: post-central gyrus. c ECoG voltage signals were streamed in 100 ms packets to update a 256 ms running buffer for online spectral pre-processing. A sample of signals from 20 channels is shown. d A Fast Fourier Transform filter was used to compute the spectral power of the 256 ms buffer, from which the high gamma (HG, 110-170 Hz) log-power was placed into a 1 s running buffer (10 feature vectors). e The running buffer was then used as time history for the recurrent neural network (RNN). f An RNN-FC (RNN-fully connected) network then predicted rest or grasp every 100 ms depending on the higher output probability. g Each classification result was stored as a vote in a 7-vote running buffer such that the number of grasp votes had to surpass a predetermined voting threshold (4-vote threshold shown) to initiate a click. A lock-out period of 1 s immediately followed every detected click to prohibit multiple clicks from occurring during the same attempted movement. Transparent images of the hand are shown to indicate the attempted grasp before a click and a relaxed configuration otherwise. h Once a click was detected, the switch scanning speller application selected the highlighted row or element within that row. Two clicks were necessary to type a letter or autocomplete a word. The example sentence shown is “the birch canoe slid on the smooth planks.”

Fig. 2. Long-term use of a fixed click detector.

a Training data was collected during 4 sessions that occurred within a period of 16 days. For each day, each sub-bar represents a separate block of training data collection (6 training blocks total). b Using the fixed detector, one block of switch scanning with the communication board was performed +21 days post-training data collection (purple). From Day +46 to Day +81, the fixed detector was used for switch scan spelling with a 7-vote threshold (blue). From Day +81 to Day +111, the fixed detector was used for switch scan spelling with a 4-vote threshold (teal). For each day, each sub-bar represents a separate spelling block of 3-4 sentences. The horizontal axis spanning both (a) and (b) represents the number of days relative to the last day of training data collection (Day 0).

Fig. 3. Switch scanning applications.

The participant was instructed to select an experimenter-cued graphical button (a) or to spell the sentence prompt (pale gray text) (b) by timing his clicks to the appropriate highlighted row or column during the switch scanning cycle. For a detailed description of (a) and (b), refer to Supplementary Figs. 8 and 9, respectively.

Fig. 4. Long-term switch scan spelling performance.

Across all subplots, triangular and circular markers represent metrics using a 7-vote and 4-vote voting threshold, respectively. a Sensitivity of click detection for each session. b True positive and false positive frequencies (TPF and FPF, which are represented by blue and green markers, respectively) were measured as detections per minute. c Latencies of grasp onset to correct algorithm detection (green markers) and on-screen click (blue markers). For each of the sessions using a 7-vote threshold, there were 284, 182, 372, 264, 451, 382, 233, 453, and 292 latency measurements, respectively. For each of the sessions using a 4-vote threshold, there were 135, 513, 591, 209, 289, 547, 511, 579, and 466 latency measurements, respectively. Mean latencies are shown as triangular or circular markers that are overlayed on top of box-and-whisker plots. The distribution of latencies across all spelling blocks in a session was used to compute the mean latency and box-and-whisker plot for that session. For each box-and-whisker plot, the median is shown as the center line, the quartiles are shown as the top and bottom edges of the box, and the whiskers are shown at 1.5 times the interquartile range. Using 7-vote and 4-vote voting thresholds, on-screen clicks happened an average of 207 ms and 203 ms, respectively after detection. Note that algorithmic detection latencies were not registered in the first six sessions. d Characters per minute (CPM) are assessed in terms of correct and wrong characters per minute (CCPM and WCPM, which are represented by blue and green markers, respectively). e Correct words per minute (CWPM).

Fig. 5. Channel importance for grasp classification.

Saliency maps for the model used online, a model using HG features from all channels except from channel 112, and a model using HG features only from channels covering cortical hand-knob are shown in (a), (c) and (e), respectively. Channels overlayed with larger and more opaque circles represent greater importance for grasp classification. White and transparent circles represent channels which were not used for model training. Mean confusion matrices from repeated 10-fold CV using models trained on HG features from all channels, all channels except for channel 112, and channels covering only the cortical hand-knob are shown in (b), (d), and (f), respectively. g Box and whisker plot showing the offline classification accuracies from 10 cross-validated testing folds using models with the above-mentioned channel subsets. Specifically, for one model configuration, each dot represents the average accuracy of the same validation fold across 20 repetitions of 10-fold CV (see Channel contributions and offline classification comparisons). Offline classification accuracies from CV-models trained on features from all channels were statistically higher than CV-models trained on features from channels only over cortical hand-knob (* P = 0.015, two-sided Wilcoxon Rank-Sum test with 3-way Holm-Bonferroni correction).