A brain-computer typing interface using finger movements

Abstract

Intracortical brain computer interfaces (iBCIs) decode neural activity from the cortex and enable motor and communication prostheses, such as cursor control, handwriting and speech, for people with paralysis. This paper introduces a new iBCI communication prosthesis using a 3D keyboard interface for typing using continuous, closed loop movement of multiple fingers. A participant-specific BCI keyboard prototype was developed for a BrainGate2 clinical trial participant (T5) using neural recordings from the hand-knob area of the left premotor cortex. We assessed the relative decoding accuracy of flexion/extension movements of individual single fingers (5 degrees of freedom (DOF)) vs. three groups of fingers (thumb, index-middle, and ring-small fingers, 3 DOF). Neural decoding using 3 independent DOF was more accurate (95%) than that using 5 DOF (76%). A virtual keyboard was then developed where each finger group moved along a flexion-extension arc to acquire targets that corresponded to English letters and symbols. The locations of these letter/symbols were optimized using natural language statistics, resulting in an approximately a 2× reduction in distance traveled by fingers on average compared to a random keyboard layout. This keyboard was tested using a simple real-time closed loop decoder enabling T5 to type with 31 symbols at 90% accuracy and approximately 2.3 sec/symbol (excluding a 2 second hold time) on average.

Keywords: Communication; Fingers; Intracortical Brain Computer Interface; Motor Cortex; Motor Decoding; Typing.

Fig. 1.

Classification analysis with confusion matrices showing accuracy of (A) five individualized finger movements and (B) fingers grouped into three groups. Movements are grouped by flexion and extension (orange colored boxes represent flexion or extension and blue box outline groups the other).

Fig. 2.

Keyboard design. (A) Unigram frequencies of all symbols from the Brown Corpus. (B) Designed keyboard. Colors indicate finger groupings, with index-middle (green) and ring-little (yellow) tied together. Keys (circles) lie along the flexion-extension movement axis of the corresponding finger, with key color indicating finger assignment. Staggered locations of keys on the same finger group allows a unique selection even though two fingers are constrained to move together. (C) Mean distance per character across sentences from Brown corpus for the optimized keyboard (black histograms) and different random keyboards (red histograms). Lines indicate Gaussian fits to each histogram. Minimum and maximum distance of keys indicated as a reference.

Fig. 3.

Closed loop typing performance. (A, B, C) Example trials showing finger positions for typing different letters. The finger group corresponding to the target key is flexed or extended until one of the fingers is over the key. The target and decoded sentences are displayed at the bottom of each figure. (D) Time per trial (including the two second hold time) vs distance of the target key from rest position. Green dots indicate successful trials and red dots indicate failure (either due to timeout of 10 seconds and selecting the wrong key). (E) Finger trajectories for the three finger groups. Black lines indicate target finger positions and yellow lines indicate the decoded finger positions. Green/red squares indicate successful/failed trials respectively. The target character is indicated above each trial. Vertical lines indicate the two second wait period at the beginning of each trial.