Evaluation of a Compact Hybrid Brain-Computer Interface System

Abstract

We realized a compact hybrid brain-computer interface (BCI) system by integrating a portable near-infrared spectroscopy (NIRS) device with an economical electroencephalography (EEG) system. The NIRS array was located on the subjects' forehead, covering the prefrontal area. The EEG electrodes were distributed over the frontal, motor/temporal, and parietal areas. The experimental paradigm involved a Stroop word-picture matching test in combination with mental arithmetic (MA) and baseline (BL) tasks, in which the subjects were asked to perform either MA or BL in response to congruent or incongruent conditions, respectively. We compared the classification accuracies of each of the modalities (NIRS or EEG) with that of the hybrid system. We showed that the hybrid system outperforms the unimodal EEG and NIRS systems by 6.2% and 2.5%, respectively. Since the proposed hybrid system is based on portable platforms, it is not confined to a laboratory environment and has the potential to be used in real-life situations, such as in neurorehabilitation.

Figure 1

Channel layout of near-infrared spectroscopy (NIRS; Ch1–Ch9) and electroencephalography (EEG; (a)) and a headgear setup on a phantom head (b). Five sources (red circles, 1–5) and three detectors (green circles, 1–3) are located around Fp1, Fpz, and Fp2. Fourteen electrodes are located at Fz, F3, F4, F7, F8, C3, C4, T7, T8, Pz, P3, P4, P7, and P8. Reference and ground electrodes are located on TP9 and TP10, respectively.

Figure 2

Timing sequence diagram of a single trial for the Stroop word-picture matching test. The whole process was done twice consecutively for congruent and incongruent tasks, which comprised a single trial. “Congruent first-incongruent later” and “incongruent first-congruent later” tasks were randomly presented. At the task presentation, the left- or right-side picture was sequentially selected. The name of either picture was displayed for 2 s. At initial mental arithmetic (MA) problem presentation, an example of a three-digit number minus a one-digit number (6 to 9) was shown instead of the name for 2 s. In a task period starting with a short beep (250 ms) and black fixation cross, subjects performed MA or baseline (BL) task if the word and picture were matched (congruent) or mismatched (incongruent), respectively. After 10 s, a rest period started with a short beep (250 ms), and a large black fixation cross was displayed at the center of the screen.

Figure 3

Grand average time-frequency analysis results for event-related (de)synchronization (ERD/ERS) in the frequency band of 4–35 Hz in frontal ((a) MA, BL, and MA-BL at Fz from left to right) and parietal areas ((b) MA, BL, and MA-BL at Pz from left to right).

Figure 4

Grand average spatial patterns for all corresponding eigenvalues: λ = (a) 0.36, (b) 0.41, (c) 0.67, and (d) 0.77. Note that the signs of the spatial patterns are irrelevant.

Figure 5

(a) Grand average time courses of changes in deoxyhemoglobin (Δ[HbR]) and oxyhemoglobin (Δ[HbO]). The log⁡(p) significance of each channel is shown horizontally at the bottom of each subplot. The red and blue solid lines correspond to MA-related and BL-related activation, respectively. A small gray shade depicts the baseline period of −5 to 0 s, and a large gray patch indicates the task period of 0 to 10 s. A solid vertical line indicates the onset of the task period. The units of the x- and y-axes are seconds and mol/L, respectively. (b) Time-dependent scalp plots of log⁡(p) significance of Δ[HbR] and Δ[HbO] based on the r-value. A color bar on the right side denotes a scale of log⁡(p) significance for both (a) and (b). The positive and negative values in the color bar indicate that MA-related activation shows higher and lower values than those of BL-related activation, respectively.

Figure 6

Grand average time-dependent NIRS (red), EEG (blue), and hybrid (HYB; black) classification accuracies. The gray shaded region shows a task period (t = 0–10 s). The red and blue asterisks below indicate the time periods in which the accuracies of HYB were significantly higher than those of NIRS (red) and EEG (blue), respectively. Error bars along with the solid lines show the standard errors.

Figure 7

Comparisons of classification performances between NIRS and HYB (blue circles) at t = 6 (a), 11 (b), and 14 s (c). At t = 6, comparison of classification performances between EEG and HYB (red circles) and between NIRS and HYB (blue circles) is provided. The three time points are selected when EEG, hybrid, and NIRS show the highest classification accuracies according to the results shown in Figure 6. Circles above the red diagonal indicate that the performance is improved by HYB compared with NIRS/EEG. Percentage values indicate the percent of subjects showing performance improvement by HYB compared with NIRS (black) or EEG (red). p values indicate significance of the performance improvement by HYB compared with NIRS (black) or EEG (red).