Nessi: an EEG-controlled web browser for severely paralyzed patients

Abstract

We have previously demonstrated that an EEG-controlled web browser based on self-regulation of slow cortical potentials (SCPs) enables severely paralyzed patients to browse the internet independently of any voluntary muscle control. However, this system had several shortcomings, among them that patients could only browse within a limited number of web pages and had to select links from an alphabetical list, causing problems if the link names were identical or if they were unknown to the user (as in graphical links). Here we describe a new EEG-controlled web browser, called Nessi, which overcomes these shortcomings. In Nessi, the open source browser, Mozilla, was extended by graphical in-place markers, whereby different brain responses correspond to different frame colors placed around selectable items, enabling the user to select any link on a web page. Besides links, other interactive elements are accessible to the user, such as e-mail and virtual keyboards, opening up a wide range of hypertext-based applications.

Figure 1

Web surfing with Nessi. Colored in-place link markers correspond to brain responses of the user, which are shown as goals in the BCI feedback window (left). After a page has loaded, the link markers are applied to a set of menu icons (top), allowing the user to choose a link, go back, scroll down the page, and use other configurable options. The number of goals and accordingly link marker colors can be increased for multiclass BCIs.

Figure 2

LSP transducer with scanning structure depicting a virtual keyboard with four output letters. The initial state is s0. Transitions are labeled with the classification answer and the logical “depth.” Internal states are marked with an “i” and states allowing for correction of errors by moving back to higher levels (back nodes) are marked with a “b.” From the current state, s0, the red states are reached with brain response 1 and the green states are reached with brain response 2.

Figure 3

Huffman-coded transducer. The probability that a node will be chosen, based on a language model (or web revisitation patterns for links), is shown inside the node. The back nodes have been omitted for clarity. Beginning at state s0, the red state is reached with brain response 1 and the green states are reached with brain response 2.

Figure 4

BCI software communicates with Nessi via a socket protocol. Text is entered into web forms or chat sites with a virtual keyboard. Switch interfaces for non-BCI users could be added. Remote supervision can be realized by starting a remote instance of Nessi, which synchronizes with the patient's display.

Figure 5

E-mail interface. (1) presents the current menu choices to the user. (2) displays incoming messages. There is an area for reading (3) and writing (4) e-mails. To allow quick selection and prevent confusion, the user chooses either the reply, compose, or next e-mail icon. The selection process is the same as for links on a web page. Addresses can be chosen from an address book created by the supervisor. Considering the fact that BCI users will generally read and write short messages, these two windows were placed next to each other, preventing the need to open new windows. E-mails are composed with a virtual keyboard.

Figure 6

Virtual keyboards for entering text into web forms, displaying the LSP transducer (center) and the Huffman-coded transducer (bottom), which are used in the simulations.

Figure 7

Number of additional brain responses to write a letter, required by the Huffman-transducer compared to the scanning transducer, as a function of the select and reject accuracies p and q (based on German letter frequencies).