Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Jul 13:15:659410.
doi: 10.3389/fnhum.2021.659410. eCollection 2021.

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Nora Hollenstein  1 , Cedric Renggli  2 , Benjamin Glaus  2 , Maria Barrett  3 , Marius Troendle  4 , Nicolas Langer  4 , Ce Zhang  2
Affiliations

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Nora Hollenstein et al. Front Hum Neurosci. .

Abstract

Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available.

Keywords: EEG; brain activity; frequency bands; machine learning; multi-modal learning; natural language processing; neural network; physiological data.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1
Figure 1
(Left) Label distribution of the 11 relation types in the relation detection dataset. (Right) Number of relation types per sentence in the relation detection dataset.
Figure 2
Figure 2
The multi-modal machine learning architecture for the EEG-augmented models. Word embeddings of dimension d are the input for the textual component (yellow); EEG features of dimension e for the cognitive component (blue). The text component consists of recurrent layers followed by two dense layers with dropout. We test multiple architectures for the EEG component (see Figure 3). Finally, the hidden states of both components are concatenated and followed by a final dense layer with softmax activation for classification (green).
Figure 3
Figure 3
EEG decoding components: (Left) The recurrent model component is analogous to the text component and consists of recurrent layers followed by two dense layers with dropout. (Right) The convolutional inception component consists of an ensemble of convolution filters of varying lengths which are concatenated and flattened before the subsequent dense layers.
Figure 4
Figure 4
Data ablation for all three word embedding types for the binary sentiment analysis task using the recurrent EEG decoding component. The shaded areas represent the standard deviations.
Figure 5
Figure 5
Data ablation for all three word embedding types for the binary sentiment analysis task using the convolutional EEG decoding component. The shaded areas represent the standard deviations.
See this image and copyright information in PMC

References

    1. Affolter N., Egressy B., Pascual D., Wattenhofer R. (2020). Brain2word: decoding brain activity for language generation. arXiv preprint arXiv:2009.04765.
    1. Alday P. M. (2019). M/EEG analysis of naturalistic stories: a review from speech to language processing. Lang. Cogn. Neurosci. 34, 457–473. 10.1080/23273798.2018.1546882 - DOI
    1. Armeni K., Willems R. M., Frank S. (2017). Probabilistic language models in cognitive neuroscience: promises and pitfalls. Neurosci. Biobehav. Rev. 83, 579–588. 10.1016/j.neubiorev.2017.09.001 - DOI - PubMed
    1. Arora S., May A., Zhang J., Ré C. (2020). Contextual embeddings: when are they worth it? arXiv preprint arXiv:2005.09117. 10.18653/v1/2020.acl-main.236 - DOI
    1. Artemova E., Bakarov A., Artemov A., Burnaev E., Sharaev M. (2020). Data-driven models and computational tools for neurolinguistics: a language technology perspective. J. Cogn. Sci. 21, 15–52.