The hybrid BCI

Abstract

Nowadays, everybody knows what a hybrid car is. A hybrid car normally has two engines to enhance energy efficiency and reduce CO2 output. Similarly, a hybrid brain-computer interface (BCI) is composed of two BCIs, or at least one BCI and another system. A hybrid BCI, like any BCI, must fulfill the following four criteria: (i) the device must rely on signals recorded directly from the brain; (ii) there must be at least one recordable brain signal that the user can intentionally modulate to effect goal-directed behaviour; (iii) real time processing; and (iv) the user must obtain feedback. This paper introduces hybrid BCIs that have already been published or are in development. We also introduce concepts for future work. We describe BCIs that classify two EEG patterns: one is the event-related (de)synchronisation (ERD, ERS) of sensorimotor rhythms, and the other is the steady-state visual evoked potential (SSVEP). Hybrid BCIs can either process their inputs simultaneously, or operate two systems sequentially, where the first system can act as a "brain switch". For example, we describe a hybrid BCI that simultaneously combines ERD and SSVEP BCIs. We also describe a sequential hybrid BCI, in which subjects could use a brain switch to control an SSVEP-based hand orthosis. Subjects who used this hybrid BCI exhibited about half the false positives encountered while using the SSVEP BCI alone. A brain switch can also rely on hemodynamic changes measured through near-infrared spectroscopy (NIRS). Hybrid BCIs can also use one brain signal and a different type of input. This additional input can be an electrophysiological signal such as the heart rate, or a signal from an external device such as an eye tracking system.

Keywords: SSVEP; brain–computer interface; event-related desynchronization; hybrid BCI; motor imagery.

Figure 1

Examples of hybrid BCIs with sequential (A,D–G) and simultaneous processing (B,C).

Figure 2

Performance measures of hand orthosis control in 10 subjects. This figure displays the errors during orthosis control (FPa; right y-axis: scale 0–1.0) and during rest (FPr; left y-axis: 0–6). The x-axis presents subjects organized from low to high FPa. Subjects’ FPas did not affect their FPrs.

Figure 3

Examples of two runs (runs #1 and #2) in one able-bodied subject (s3) over several minutes each. The lower traces of runs #1 and #2 display the four-step sequence of opening/closing the SSVEP-based orthosis with two 60-s breaks (grey shaded). The upper traces of runs #1 and #2 show the ERS-based switch operation (black bars indicate switch opened). The four-steps of orthosis opening (from left to right) are displayed in the bottom panel.

Figure 4

From top to bottom: Position trace of the switch (grey areas mark closed switch position) and four-step SSVEP-based orthosis control trace (grey areas indicate 60-s resting periods) of run#2; position trace of brain switch and orthosis control of run#3; level of oxyhemoglobin (HbO₂) concentration of run#3; views of prefrontal optodes and bipolar occipital EEG electrode placements. FPs of switch (FP) and SSVEP-based orthosis control (FPa, FPr) are indicated. Note the HbO₂ peaks associated with the intended mental tasks.

Figure 5

Raw EEG, heart rate and time course of the logarithmic band power (15–19 Hz), enlarged from a 10-s time window (lower panel, left), and averaged logarithmic beta power (mean ± SD) together with synchronous averaged HR response (mean ± SD, lower panel, right). Remarkably, the HR increase starts some seconds before the band power enhancement. Modified from Pfurtscheller et al. (2008b).

Figure 6

Beta power and HR changes during self-paced motor imagery. This Figure shows logarithmic beta power with online detected output signals (vertical lines) during mental practice in virtual environment (for details see Pfurtscheller et al., 2008b), HR and first derivative of HR (dHR). The dHR time course shows that the detection of foot motor imagery with the HR correlates well with EEG detection and revealed six TPs, one FP and two FNs.

Figure 7

Prosthetic hand with four mounted LEDs (A), examples of respiratory signals (Resp), heart beat-to-beat intervals (RRI) measured in seconds and first derivative of RRI (dRRI) during intentional (B) and non-intentional control (C). Two motion sequences (O, R, L, C) and the threshold are indicated. Modified from Scherer et al. (2007).

Figure 8

For condition “easy”, accuracy of the BCI based solution was only slightly lower (88%) compared to the long dwell times (DTL, 93%). Short dwell times (DTS) resulted in the lowest mean accuracy (83.3%). Remarkably, the BCI achieves the best results in accuracy for the condition “difficult” (78.7%), but only the difference to the short dwell time (51.1%) was significant.

Figure 9

Example of discrimination time courses (off-line classification accuracy) from two different studies with visual cue-based right and left hand motor imagery. In one study (Pfurtscheller et al., 2008c) subjects with BCI experience took part, whereas the other study used naïve subjects (for details see “Simultaneous ERD/SSVEP BCI to Improve Accuracy”; Allison et al., 2010). (A,B) Display the discrimination accuracy of two subjects from the group of BCI experienced subjects, and (C) displays superimposed accuracy time courses of five subjects of the naïve group. In all examples an early discrimination peak ∼1 s after visual cue onset is visible. Cue duration is indicated by the grey area.