Measuring Mental Workload with EEG+fNIRS

Abstract

We studied the capability of a Hybrid functional neuroimaging technique to quantify human mental workload (MWL). We have used electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) as imaging modalities with 17 healthy subjects performing the letter n-back task, a standard experimental paradigm related to working memory (WM). The level of MWL was parametrically changed by variation of n from 0 to 3. Nineteen EEG channels were covering the whole-head and 19 fNIRS channels were located on the forehead to cover the most dominant brain region involved in WM. Grand block averaging of recorded signals revealed specific behaviors of oxygenated-hemoglobin level during changes in the level of MWL. A machine learning approach has been utilized for detection of the level of MWL. We extracted different features from EEG, fNIRS, and EEG+fNIRS signals as the biomarkers of MWL and fed them to a linear support vector machine (SVM) as train and test sets. These features were selected based on their sensitivity to the changes in the level of MWL according to the literature. We introduced a new category of features within fNIRS and EEG+fNIRS systems. In addition, the performance level of each feature category was systematically assessed. We also assessed the effect of number of features and window size in classification performance. SVM classifier used in order to discriminate between different combinations of cognitive states from binary- and multi-class states. In addition to the cross-validated performance level of the classifier other metrics such as sensitivity, specificity, and predictive values were calculated for a comprehensive assessment of the classification system. The Hybrid (EEG+fNIRS) system had an accuracy that was significantly higher than that of either EEG or fNIRS. Our results suggest that EEG+fNIRS features combined with a classifier are capable of robustly discriminating among various levels of MWL. Results suggest that EEG+fNIRS should be preferred to only EEG or fNIRS, in developing passive BCIs and other applications which need to monitor users' MWL.

Keywords: cognitive state monitoring; electroencephalography (EEG); functional near-infrared spectroscopy (fNIRS); human mental workload; machine learning; multi-modal brain recording; n-back.

Figure 1

Schematic illustration of the letter n-back task for n ϵ {0, 1, 2, 3}.

Figure 2

Experimental design for the letter n-back task. Each session includes the Instruction, task, and rest blocks.

Figure 3

(a) EEG+fNIRS recording setup. Subject interaction with the computer, synchronization of EEG and fNIRS signal, recording of EEG and fNIRS signals, and data transmission to the acquisition platform. (b) Coronal view of the subject showing the close view of the placement fNIRS optodes and EEG electrodes. (c) Topographical view of fNIRS sources (S_i, black) and detectors (D_i, red) and EEG electrodes (green). Each pair of source and detector separated by 3 cm creates a channel (CH_i). We used the signals from F7, Fpz, and F8.

Figure 4

Topographic view of EEG electrodes showing neighborhood pattern for Laplacian spatial filtering. Inward arrows to each node indicate the corresponding neighbors used for spatial filtering.

Figure 5

Sample preprocessed EEG+fNIRS data for one of the subjects. Vertical dashes separate different n-back task and rest blocks. (a) Concentration changes of oxy-hemoglobin (red curve) and deoxy-hemoglobin (blue) for channel 17. (b) EEG Time-frequency map of the channel O2.

Figure 6

Four different epoch styles based on length of windows. The task and rest blocks are divided into (A) 5, (B) 10, (C) 20, and (D) 25 s windows (w_i).

Figure 7

Behavioral performance of the subjects during task conditions of increasing difficulty, showing response accuracy (red) and response time (black). Error bars indicate the standard deviation of inter-subject variability. Asterisks indicate statistical significance derived from a two-way ANOVA comparison of each two response accuracy (red) or response time (black) (^*p < 0.05, ^**p < 0.001, ^***p < 0.0001).

Figure 8

Grand block average of normalized HbO (red) and HbR (blue) during (a) 0-back, (b) 1-back, (c) 2-back, (d) 3-back, (e) rest. The thick curves show the average over all channels and subjects. The shaded area indicates the standard deviation of inter-subject variability. Grand block average of HbO (f) and HbR (g) for rest (dashed curves) and task (solid). Increasing thickness of solid curves corresponds to increasing task difficulty from 0- to 3-back. AU means arbitrary units.

Figure 9

Grand block average of normalized features from 5 s windows: (a) PSD (delta, theta, alpha bands) of channel O2. (b) HbO/R Amp. for channel 10. (c) NVO (delta, theta, alpha bands) for channel 10. (d) NVR features (delta, theta, alpha bands) for channel 10. Shaded areas indicate the standard deviation of inter-subject variability.

Figure 10

Accuracy of types of features in classifying rest v 3-back with 5 s feature windows. The error bars indicate the standard deviation of inter-subject variability. The union of neurovascular features is abbreviated as NV. Features are extracted from different systems: EEG (gray bars), fNIRS (red bars), and Hybrid (green bars).

Figure 11

(a) Accuracy and (b) cumulative sum of R² for EEG (black), fNIRS (red), and Hybrid (green) systems as a function of system size. Mean and standard deviation over subjects are indicated by the solid curves and shaded areas, respectively. The classification task was rest v 3-back and feature window size was 5 s.

Figure 12

Accuracy of the rest v 3-back classification as a function of window size for EEG (gray), fNIRS (red), and Hybrid (green) systems. Error bars indicate the standard deviation of inter-subject variability.