Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Abstract

Semantic knowledge about individual entities (i.e., the referents of proper names such as Jacinta Ardern) is fine-grained, episodic, and strongly social in nature, when compared with knowledge about generic entities (the referents of common nouns such as politician). We investigate the semantic representations of individual entities in the brain; and for the first time we approach this question using both neural data, in the form of newly-acquired EEG data, and distributional models of word meaning, employing them to isolate semantic information regarding individual entities in the brain. We ran two sets of analyses. The first set of analyses is only concerned with the evoked responses to individual entities and their categories. We find that it is possible to classify them according to both their coarse and their fine-grained category at appropriate timepoints, but that it is hard to map representational information learned from individuals to their categories. In the second set of analyses, we learn to decode from evoked responses to distributional word vectors. These results indicate that such a mapping can be learnt successfully: this counts not only as a demonstration that representations of individuals can be discriminated in EEG responses, but also as a first brain-based validation of distributional semantic models as representations of individual entities. Finally, in-depth analyses of the decoder performance provide additional evidence that the referents of proper names and categories have little in common when it comes to their representation in the brain.

Keywords: EEG; brain decoding; categories; distributional semantics; individual entities; language models; proper names.

Figure 1

Experimental design and hierarchical stimuli organization. Our set of stimuli was organized symmetrically, so that exactly the same number of stimuli was present for each coarse- and fine-grained category—as shown in (A). Stimuli included both proper names of individual entities and their fine-grained categories. In (B) we present the experimental paradigm. Each stimulus was projected in isolation on the screen for 750 ms, and subjects then had to mentally visualize its referent while a cross remained on screen for 1,000 ms. Participants then answered a question, involving the fine- or coarse-grained level of categorization.

Figure 2

Clustering results on word vectors, with various possible assignments of categorical labels. We test the performance of each model on a completely unsupervised clustering task. In this evaluation, we measure to what extent word vectors can be clustered according to their category, evaluating separately coarse-grained and fine-grained categories. The evaluation metric that we used, Adjusted Rand Index, involves a correction for chance, so results above 0.0 show some sensitivity to categorical structure. We report results for all possible combinations of stimuli: clustering using only the vectors for individuals; only the vectors for categories; both categories and individual entities; people and places separately. Most models perform above chance in all clustering tasks.

Figure 3

Classification of coarse-grained categories from individuals. We run a time-resolved binary classification on the EEG data. For each time point, we learn to classify evoked responses according to their coarse semantic category (either person or place). We control for word length employing the test sets where the correlation between labels and stimulus length is lowest (see Section 3.4). The green line is the average of results across the 33 participants, and the shaded areas correspond to the standard error of the mean. The random baseline is at 0.5, given that this is a binary classification problem, and is represented by a dotted horizontal line; we also plot as dotted vertical lines the time-points when stimuli appear and disappear. Statistically significant points (p < 0.05 corrected for multiple comparisons by TFCE; see Section 3.4) are reported both on the averaged lines and, to make them easier to read, below the x axis.

Figure 4

Classification of coarse-grained categories, transferring information from individual entities to categories. Here we report time-resolved classification scores, following the same structure as Figure 3. However, in this case we look explicitly at whether discriminative representational information is shared across individuals and entities, training on individuals, and testing on categories. The results show that this is not the case: scores go past the random baseline only after the stimulus disappears, and never reach statistical significance.

Figure 5

Classification of fine-grained categories from individuals. We run a time-resolved multi-class classification analysis, trying to decode at each time point the fine-grained semantic category of the stimulus. Figure structure is the same as in Figure 3, but here the random baseline is at 0.125, since there are eight possible categories (four for people and four for places). Scores are statistically significant between 300 and 400 ms, and from 800 to 1,100 ms.

Figure 6

Classification of fine-grained categories, transferring information from individual entities to categories. This figure is the equivalent of Figure 4, just referring to the multi-class classification task involving fine-grained categories as labels. The random baseline is set at 0.125, because of the presence of eight possible classes. Notice that decoding scores follow a very different course with respect to coarse-grained categories. Statistical significance is never reached, confirming that little discriminative semantic information is shared between responses to individuals and categories.

Figure 7

Classification of fine-grained categories, separately for people and places, considering individuals only. We plot classification scores against time, as reported above in Figures 3–6, with the exception that here we restrict our analyses to evoked responses for either people or places. Since there are only four possible classes, random baseline is at 0.25. Interestingly, for people results never reach significance, suggesting that people categories are hard to decode from EEG data. Instead, when decoding fine-grained semantic categories of places, results are strikingly different: classification reaches statistical significance between 300 and 400 ms.

Figure 8

Decoding to word vectors, considering only individual entities. We report the distribution of the scores for 33 subjects when predicting word vector dimensions from EEG data. In this case, we employ evoked responses and word vectors for individual entities. Evaluation is carried out using the leave-two-out methodology proposed in Mitchell et al. (2008). For each computational model, we plot the distributions as violin plots, where the white dots indicate the model average score, and gray bars are used for the 95% confidence interval. We run a one-tailed Wilcoxon statistical significance test against the chance baseline of 0.50. Stars indicate the resulting p-value. In general, results are well above random performance, providing evidence that it is possible to distinguish representations of individual entities in the brain, by exploiting computational models of language. Also, contextualized models perform better than their static counterpart.

Figure 9

Breakdown of results when decoding to word vectors, considering both individuals and categories. Making the most of the leave-two-out evaluation methodology, which allows to split results according to the semantic and ontological category of the two words, we break down the results for the model with the highest average score, which is BERT large. Each violin plots the distribution of scores for a given combination of categories across subjects. Scores are statistically significant in all cases, except for the case when only categories are used. There, only one set of scores is statistically significant at p < 0.05, and the 95% confidence interval always include the baseline, indicating that performance is not reliably above chance. Confirming that individual entities and categories are easiest to discriminate, scores improve dramatically when an individual entity and a category are left out for testing.