A 36-Class Bimodal ERP Brain-Computer Interface Using Location-Congruent Auditory-Tactile Stimuli

Abstract

To date, traditional visual-based event-related potential brain-computer interface (ERP-BCI) systems continue to dominate the mainstream BCI research. However, these conventional BCIs are unsuitable for the individuals who have partly or completely lost their vision. Considering the poor performance of gaze independent ERP-BCIs, it is necessary to study techniques to improve the performance of these BCI systems. In this paper, we developed a novel 36-class bimodal ERP-BCI system based on tactile and auditory stimuli, in which six-virtual-direction audio files produced via head related transfer functions (HRTF) were delivered through headphones and location-congruent electro-tactile stimuli were simultaneously delivered to the corresponding position using electrodes placed on the abdomen and waist. We selected the eight best channels, trained a Bayesian linear discriminant analysis (BLDA) classifier and acquired the optimal trial number for target selection in online process. The average online information transfer rate (ITR) of the bimodal ERP-BCI reached 11.66 bit/min, improvements of 35.11% and 36.69% compared to the auditory (8.63 bit/min) and tactile approaches (8.53 bit/min), respectively. The results demonstrate the performance of the bimodal system is superior to each unimodal system. These facts indicate that the proposed bimodal system has potential utility as a gaze-independent BCI in future real-world applications.

Keywords: ANOVA; BCI; BLDA; ERP; HRTF; auditory; bimodal stimulus; electro-tactile; gaze-independent; location-congruent.

Figure 1

Illustration of the character distribution of the 36-class brain-computer interface (BCI) system and the bimodal stimulus paradigm. In subfigure (a), the prompted character “J”, as an example, is marked red. In subfigure (b), the stimuli were delivered from corresponding location. Each yellow broken line represents a pair of electrical tactile electrodes placed at waist level. The arrows represent the directions of auditory stimuli delivered from headphones. The numbers identify the stimulus codes (location-congruent bimodal stimuli from “1” to “6”).

Figure 2

Illustration of electro-tactile impulse stimuli.

Figure 3

Stimulus sequence generation.

Figure 4

Configuration of electrode locations.

Figure 5

Flow diagram of Jumpwise regression algorithm. The value pU is the p-value of the partial F-test statistics of the model containing all features of the selected channel set (SC) against the features in SC except those belonging to the unselected channel set (UC), and the value pS is the p-value of the partial F-test statistics of the model containing all features of SC against the features in SC except those belonging to SC. The p-value thresholds for adding and removing channels were set to 0.1 and 0.15, respectively.

Figure 6

Offline ITR (a) and classification accuracy (b) of auditory, tactile and bimodal BCI using the first ten runs of each session. Error bars denote the standard deviation.

Figure 7

Classification accuracy comparisons among the six directions of auditory (a), tactile (b), and bimodal (c) approaches. The green line divided the stimulus area into the right-posterior part (“4”, “5” and “6”) and the left-anterior part (“1”, “2” and “3”) for the ease of analyzing significance.

Figure 8

The results of channel selection for each subject. (a–c) denotes the eight optimal channels of each subject for the three modalities; (d–f) represents the total number of selected channels on the scalp for the twelve subjects. The color bar from dark blue to dark red indicates the summed number of channel selections for each electrode from zero to eight times, respectively.