EEG Negativity in Fixations Used for Gaze-Based Control: Toward Converting Intentions into Actions with an Eye-Brain-Computer Interface

Abstract

We usually look at an object when we are going to manipulate it. Thus, eye tracking can be used to communicate intended actions. An effective human-machine interface, however, should be able to differentiate intentional and spontaneous eye movements. We report an electroencephalogram (EEG) marker that differentiates gaze fixations used for control from spontaneous fixations involved in visual exploration. Eight healthy participants played a game with their eye movements only. Their gaze-synchronized EEG data (fixation-related potentials, FRPs) were collected during game's control-on and control-off conditions. A slow negative wave with a maximum in the parietooccipital region was present in each participant's averaged FRPs in the control-on conditions and was absent or had much lower amplitude in the control-off condition. This wave was similar but not identical to stimulus-preceding negativity, a slow negative wave that can be observed during feedback expectation. Classification of intentional vs. spontaneous fixations was based on amplitude features from 13 EEG channels using 300 ms length segments free from electrooculogram contamination (200-500 ms relative to the fixation onset). For the first fixations in the fixation triplets required to make moves in the game, classified against control-off data, a committee of greedy classifiers provided 0.90 ± 0.07 specificity and 0.38 ± 0.14 sensitivity. Similar (slightly lower) results were obtained for the shrinkage Linear Discriminate Analysis (LDA) classifier. The second and third fixations in the triplets were classified at lower rate. We expect that, with improved feature sets and classifiers, a hybrid dwell-based Eye-Brain-Computer Interface (EBCI) can be built using the FRP difference between the intended and spontaneous fixations. If this direction of BCI development will be successful, such a multimodal interface may improve the fluency of interaction and can possibly become the basis for a new input device for paralyzed and healthy users, the EBCI "Wish Mouse."

Keywords: assistive technology; brain-computer interfaces; detection of intention; eye tracking; gaze interaction; human-computer interfaces; slow cortical potentials; stimulus-preceding negativity.

Figure 1

A direct way to convey your desire to a computer: just look! When using a monitor, we normally look at a screen button or a link before clicking on them (upper panel). Between looking and clicking, we take the mouse (if we are not holding it all time), locate the cursor, move the cursor to the button or link, check if it reached the target. Gaze fixation at a given monitor location, however, can be promptly recognized with an eye tracker. If our intention to click would be recognized with some technology based on brain signal analysis (lower panel, blue line), computer control could be obtained without the above listed range of motor and sensory activities. (Generally, detection of gaze fixations alone is not enough due to the Midas touch problem, see text for details).

Figure 2

An example screenshot of the EyeLines display. The participants moved the color balls in order to construct lines from balls of the same color. Each move was made with gaze fixations only. A participant could make any number of spontaneous fixations (S) without any visible effect; however, their time was recorded. A dwell on the button (1) lead to appearing a ball in the button position, indicating that “control is on.” After this, a dwell on any ball in the game board (2) led to its “selection” (a frame appeared around it), and then the ball could be placed to a new position by a dwell on a free cell (3). (The digits 1, 2, 3, and the letter “S” were not a part of the actual display and were added to the screenshot for illustrative purposes).

Figure 3

Butterfly plots showing grand average (n = 8) superimposed channels with fixation-related potentials (FRPs, upper panels) and corresponding electrooculogram (EOG, lower panels) for fixations on button. The signals were low-pass filtered at 7 Hz and baseline corrected (high-pass filtering was not used). Note the similarity between the waveforms for left and right button positions (in the left and in the right, respectively) within most of the fixation interval. Zero millisecond corresponds to the beginning of fixation. The baseline interval (200–300 ms) is shown by dark gray bars. The light gray bars show 400–500 ms interval from which the EEG amplitudes were averaged to obtain estimates of the trend over the fixation used in the subsequent analysis (Figures 4, 5). Dwell time threshold was 500 ms (marked with the vertical lines in the right edges of the light gray bars).

Figure 4

Grand average (n = 8) amplitude topography of the negative potential developed within different types of fixations and different dwell time thresholds. L, left button position; R, right button position (note that the switch-on button was “clicked” not manually but by gaze fixations only). Amplitude estimates were obtained by averaging over 400–500 ms interval relative to fixation start (baseline 200–300 ms). Small white circles highlight the electrodes which were used in statistical analysis (POz, PO3, and PO4). Note that non-controlling fixations were rare in Dwell 1000 ms condition (see Table 1), thus the corresponding maps can be considered only as a rough estimate.

Figure 5

Grand mean (n = 8) values and 95% confidence intervals of the negative potential at POz developed within different types of fixations, using different dwell time thresholds and button positions. Amplitude estimates were obtained by averaging over 400.500 ms interval relative to fixation start (baseline 200. 300 ms). Note that No-Control fixations were rare in Dwell 1000 ms condition (see Table 1), thus the average for these data can be considered only as a rough estimate.

Figure 6

Comparing different dwell time thresholds and button positions: grand average (n = 8) and individual FRP waveforms observed at POz in Button fixations. Fixation start and the 500 ms dwell time threshold are denoted by vertical lines. Filtering and baseline as in Figure 3.

Figure 7

Comparing different types of fixations: grand average (n = 8) and individual FRP waveforms observed at POz with 500 ms dwell time threshold. Here, left and right button data were averaged. Fixation start and the dwell time threshold are denoted by vertical lines. Filtering and baseline as in Figure 3.

Figure 8

Classifier sensitivity on the test data. Grand mean (n = 8) values and 95% confidence intervals for different classifiers, different types of fixations and different dwell time thresholds. The target (controlling) class in the train data was either button fixations (Trainset 1) only or all controlling fixations (Trainset 2).