Simultaneous EEG and fNIRS recordings for semantic decoding of imagined animals and tools

Abstract

Semantic neural decoding aims to identify which semantic concepts an individual focuses on at a given moment based on recordings of their brain activity. We investigated the feasibility of semantic neural decoding to develop a new type of brain-computer interface (BCI) that allows direct communication of semantic concepts, bypassing the character-by-character spelling used in current BCI systems. We provide data from our study to differentiate between two semantic categories of animals and tools during a silent naming task and three intuitive sensory-based imagery tasks using visual, auditory, and tactile perception. Participants were instructed to visualize an object (animal or tool) in their minds, imagine the sounds produced by the object, and imagine the feeling of touching the object. Simultaneous electroencephalography (EEG) and near-infrared spectroscopy (fNIRS) signals were recorded from 12 participants. Additionally, EEG signals were recorded from 7 other participants in a follow-up experiment focusing solely on the auditory imagery task. These datasets can serve as a valuable resource for researchers investigating semantic neural decoding, brain-computer interfaces, and mental imagery.

Fig. 1

Illustration of a single concept trial in Datasets 1 and 2. In Dataset 1, the order of mental tasks was randomized across blocks. Based on.

Fig. 2

The frontal and temporal montages used for fNIRS data acquisition in Dataset 1 alongside the joint EEG system with 64 electrodes, following the international 10-20 system. The fNIRS sources (depicted as circles) and detectors (depicted as squares) were positioned according to the 10-5 system, forming channels (depicted as small circles) located between the sources and detectors.

Fig. 3

The frequency of mental task appearances as the first, second, third, and fourth tasks in the shared order of mental tasks for participants 2 to 12 in Dataset 1.

Fig. 4

Examples of average EEG ERPs with respect to the image presentation for a single participant from Dataset 1 and 2. N_ave represents the number of trials over which the channel data were averaged.

Fig. 5

The grand average visual EEG ERP with respect to the image presentation for each participant. The black bold line indicates the average across participants.

Fig. 6

The average fNIRS power spectrum for each participant in Dataset 1.

Fig. 7

Examples of fNIRS ERPs with respect to the image presentation for two participants in Dataset 1, the first one with the frontal montage and the second one with with the temporal montage. N_ave represents the number of trials over which the channel data were averaged.