Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

Abstract

The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO₂) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO₂, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO₂-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.

Keywords: Cognitive tasks; DL networks; Decision fusion; EEG; HbO2; HbR; fNIRS.

Fig. 1

The typical architecture of CNN

Fig. 2

Structure and operation of a single LSTM cell (* = pointwise multiplication; + = pointwise addition; σ = sigmoid function; $\int$ =  tanh function)

Fig. 3

Structure and operation of a single GRU cell (X = hadamard product; + = pairwise addition; σ = sigmoid function; $\int$  =  tanh function)

Fig. 4

Tasks performed by subjects in a dataset 01 and b dataset 02

Fig. 5

Proposed methodology of decision fusion for a dataset 01 and b dataset 02

Fig. 6

Architecture of CNN classification network

Fig. 7

Architecture of LSTM classification network

Fig. 8

Architecture of GRU classification network

Fig. 9

Architecture of CNN–LSTM classification network

Fig. 10

Architecture of CNN–GRU classification network

Fig. 11

Architecture of LSTM–GRU classification network

Fig. 12

Architecture of CNN–LSTM–GRU classification network

Fig. 13

ROC curves for a decision fusion, b EEG, c HbO₂ and d HbR in dataset 01

Fig. 14

ROC curves for a decision fusion, b EEG and c fNIRS in dataset 02

Fig. 15

Performance of different modalities in dataset 01: a accuracy; b AUC

Fig. 16

Performance of different modalities in dataset 02: a accuracy; b AUC

Fig. 17

Performance of different classifiers in dataset 01: a accuracy; b AUC

Fig. 18

Performance of different classifiers in dataset 02: a accuracy; b AUC