Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface

Abstract

In a conventional brain-computer interface (BCI) system, users perform mental tasks that yield specific patterns of brain activity. A pattern recognition system determines which brain activity pattern a user is producing and thereby infers the user's mental task, allowing users to send messages or commands through brain activity alone. Unfortunately, despite extensive research to improve classification accuracy, BCIs almost always exhibit errors, which are sometimes so severe that effective communication is impossible. We recently introduced a new idea to improve accuracy, especially for users with poor performance. In an offline simulation of a "hybrid" BCI, subjects performed two mental tasks independently and then simultaneously. This hybrid BCI could use two different types of brain signals common in BCIs - event-related desynchronization (ERD) and steady-state evoked potentials (SSEPs). This study suggested that such a hybrid BCI is feasible. Here, we re-analyzed the data from our initial study. We explored eight different signal processing methods that aimed to improve classification and further assess both the causes and the extent of the benefits of the hybrid condition. Most analyses showed that the improved methods described here yielded a statistically significant improvement over our initial study. Some of these improvements could be relevant to conventional BCIs as well. Moreover, the number of illiterates could be reduced with the hybrid condition. Results are also discussed in terms of dual task interference and relevance to protocol design in hybrid BCIs.

Fig. 1

Overview of all eight analyses in this study.

Fig. 2

(a) Computer screen and flickering LEDs (below screen) used in this study. (b) Timing of each trial.

Fig. 3

(a) Results from the post hoc test in analysis 1. Shaded boxes mark significant differences, and the “<” or “>” signs indicate if the condition on the left was smaller or greater than the condition on top. White boxes indicate non-significant differences. (b) Results from the post hoc test in analysis 4.

Fig. 4

Time course of classification accuracies obtained by a running classifier in 0.125 ms segments. The plots show results for the three different smoothing windows 0.5 s, 1.0 s, and 1.5 s (from left to right). The three conditions ERD (red, dotted), SSVEP (blue, dashed), and hybrid (magenta, solid) are shown; in addition, the artificial hybrid condition is plotted in green (dash-dotted).

Fig. 5

Left: Error bars indicating the ERD performance in the ERD-only and hybrid runs (analysis 4). The crosses mark the mean classification accuracy, while the bars denote the 95% confidence intervals around the means. Right: Classification performance in the three conditions ERD-only, SSVEP-only, and hybrid (analysis 6).

Fig. 6

Grand average ERDS maps of the three conditions ERD (C3, C4), SSVEP (O1, O2), and hybrid (C3, C4, O1, O2). The vertical line at second 2 denotes the occurrence of the cue. For a detailed description, see main text.