Neural Point-and-Click Communication by a Person With Incomplete Locked-In Syndrome

Abstract

A goal of brain-computer interface research is to develop fast and reliable means of communication for individuals with paralysis and anarthria. We evaluated the ability of an individual with incomplete locked-in syndrome enrolled in the BrainGate Neural Interface System pilot clinical trial to communicate using neural point-and-click control. A general-purpose interface was developed to provide control of a computer cursor in tandem with one of two on-screen virtual keyboards. The novel BrainGate Radial Keyboard was compared to a standard QWERTY keyboard in a balanced copy-spelling task. The Radial Keyboard yielded a significant improvement in typing accuracy and speed-enabling typing rates over 10 correct characters per minute. The participant used this interface to communicate face-to-face with research staff by using text-to-speech conversion, and remotely using an internet chat application. This study demonstrates the first use of an intracortical brain-computer interface for neural point-and-click communication by an individual with incomplete locked-in syndrome.

Keywords: ALS; assistive technology; paralysis; spinal cord injury; stroke; text entry.

Figure 1

Virtual keyboard designs. (a) Radial Keyboard screen capture of the participant typing the word “quick”. Left: the keys containing the “q” and “u” have already been entered, as denoted by the “MNOPQ” and “RSTU” text shown below “the” in the word editor (and above the red asterisk). The red trace (overlaid onto the screen capture, not visible during actual system use) shows the trajectory used to select the “IJKL” key to enter the “i” in quick (the blue dot shows the neurally-clicked selection location). Middle: after entering the “i”, the desired word (“quick”) appeared in the word list; the participant selected the green right facing arrow key to bring the word completions onto the radial keys. Right: the participant selected the word “quick” from the word completions to enter the word in the editor. (b) The participant typed the word “quick” with the QWERTY keyboard. Left: the participant moved to and selected “q”. Right: the participant moved across the screen to the word completion to select “quick”.

Figure 2

Continuous (Kalman Filter) and State decoders (LDA) used for neural point-and-click control for each of the 3 copy-spelling sessions. (a) Kalman filter decoders: each arrow shows the tuning of a channel; the direction represents its preferred direction, and the amplitude represents its modulation depth in Hz. Each gray ring represents a 2 Hz increment in modulation depth. (b) State “click” decoders: LDA modeled Gaussian distributions for movement (dashed) and click (solid) classes. Gray bar shows the decision boundary (threshold) chosen to minimize false clicks. A click is decoded when the log-likelihood exceeds the threshold for 3 consecutive samples (300 msec).

Figure 3

Pooled performance results from the copy-spelling tasks. Top Row: performance in the “keyboard” task. Bottom Row: performance in the “quick fox” task. Left column: percent correct (accuracy). Middle column: keystrokes per minute. Right column: correct characters per minute. Bars = mean performance across the 6 blocks from all 3 sessions. Data points show individual block performance for the 6 repetitions of typing each phrase for each task across the 3 sessions. Data points are offset horizontally, representing the temporal order of their occurrence (left to right; circles, diamonds, and squares correspond to trial days 1895, 1918, and 1925, respectively). * = p < 0.05, ** = p < 0.0001 (one-tailed paired t-test).

Figure 4

Question/Answer session results. Top: typing performance for each answer the participant gave. Bottom: questions asked and S3's responses. To provide some context for S3's responses: Question 1: due to her competitive nature (and great sense of humor), S3 decided to type the copy spelling task phrase one more time to test her skill. Question 3: S3's favorite beverage during sessions was a cinnamon latte. It is unknown whether lattes improve neural control. Question 4: Abe was the first clinical technician S3 worked with as a part of the clinical trial. Question 6: S3 successfully controlled an assistive robotic arm and a prosthetic arm a few days before this final research session. Question 7: A final statement from S3 using the investigational BrainGate Neural Interface System.

Figure 5

Keystrokes per character (KSPC). KSPC comparison between QWERTY and Radial Keyboards for the “keyboard” task, “quick fox” task, and typing the entire software dictionary (58,000+ words). KSPC is defined (on average) as the theoretical number of keystrokes needed to enter a single character of text, assuming 100 percent accuracy. Due to the extra keystroke needed to make word selections and the weaker prediction power inherit in ambiguous text prediction compared to the QWERTY keyboard, KSPC is lower for the QWERTY keyboard. For text entry interfaces, a lower KSPC is better. In the “keyboard” task, word completion was not used, so the KSPC for QWERTY = 1, while it is 1.1 for Radial, due to the extra keystroke needed to select the word “keyboard”. In the “quick fox” task, all of the words are very common, so the KSPC is well below 1 for QWERTY. It is at 1 for the Radial keyboard in the “quick fox” task, offsetting some of the extra required keystrokes by being able to select the words as soon as they appear in the word list, which often occurs before all characters have been typed.