Structural similarities between brain and linguistic data provide evidence of semantic relations in the brain

Abstract

This paper presents a new method of analysis by which structural similarities between brain data and linguistic data can be assessed at the semantic level. It shows how to measure the strength of these structural similarities and so determine the relatively better fit of the brain data with one semantic model over another. The first model is derived from WordNet, a lexical database of English compiled by language experts. The second is given by the corpus-based statistical technique of latent semantic analysis (LSA), which detects relations between words that are latent or hidden in text. The brain data are drawn from experiments in which statements about the geography of Europe were presented auditorily to participants who were asked to determine their truth or falsity while electroencephalographic (EEG) recordings were made. The theoretical framework for the analysis of the brain and semantic data derives from axiomatizations of theories such as the theory of differences in utility preference. Using brain-data samples from individual trials time-locked to the presentation of each word, ordinal relations of similarity differences are computed for the brain data and for the linguistic data. In each case those relations that are invariant with respect to the brain and linguistic data, and are correlated with sufficient statistical strength, amount to structural similarities between the brain and linguistic data. Results show that many more statistically significant structural similarities can be found between the brain data and the WordNet-derived data than the LSA-derived data. The work reported here is placed within the context of other recent studies of semantics and the brain. The main contribution of this paper is the new method it presents for the study of semantics and the brain and the focus it permits on networks of relations detected in brain data and represented by a semantic model.

Figure 1. Conditional probability density estimates (shown as a heat map) computed from the confusion matrix resulting from the classification of 640 brain wave samples from S18 for the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

This table was taken from .

Figure 2. Semantic similarity matrix (shown as a heat map) derived from WordNet for the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia} using senses relevant to the geography of Europe.

Figure 3. Hierarchical cluster tree computed from the WordNet-based semantic similarity matrix for {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia} given in Figure 2.

Figure 4. Similarity scores computed using LSA for the words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia} shown as a heat map.

Figure 5. Hierarchical cluster tree computed from the LSA scores of similarity for {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia} given in Figure 4.

Figure 6. WordNet-based and LSA-based semantic similarities and EEG conditional probability estimates for London relative to London, Moscow, Paris, north, south, east, west, Germany, Poland, and Russia.

Data taken from Figure 1, Figure 2, and Figure 4.

Figure 7. Joint invariant partial order for London relative to the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

Brain data derived from FIGURE 1 and semantic data from the WordNet data in Figure 2.

Figure 8. Joint invariant partial order for London relative to the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

Brain data derived from Figure 1 and the LSA data from Figure 4.

Figure 9. Conditional probability density estimates (shown as a heat map) from a new single-trial classification of the brain data (S18) for {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

Figure 10. Ordinal relation of similarity differences represented as a directed acyclic graph CG^B _London for London relative to the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}, derived from the probability density estimates for the brain data in Figure 1.

Figure 11. Ordinal relation of similarity differences represented as a directed acyclic graph CG^S _London for London relative to the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}, derived from the WordNet data in Figure 2.

Figure 12. Single-trial classification rates over 120 trials for the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

640 samples (S18) classified into 10 classes. Chance accuracy is shown by the thick solid line.

Figure 13. Probability that a test sample is classified as Moscow given that it is London (dash), Paris (double) or Germany (solid) over 4 sets of 30 single-trial classifications (S18) of the words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

Figure 14. Hierarchical cluster trees computed from the accumulated average conditional probability density estimates obtained after 22 and 23 classifications of 640 brain wave samples (S18) for {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia }.

Figure 15. Hierarchical cluster tree computed from the accumulated conditional probability density estimates obtained after 2 classifications of 640 brain wave samples (S18) for {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia}.

For all successive runs, the accumulated similarity tree did not change.

Figure 16. Significant structural similarities (partial orders of similarity differences invariant between the brain data and the linguistic data) for the set of words {London, Moscow, Paris, north, south, east, west, Germany, Poland, Russia} from 60 single-trial classifications for each participant.

Figure 17. Significant structural similarities (partial orders of similarity differences invariant between the brain data and the linguistic data) for the set of words {Paris, Vienna, Athens, north, south, east, west, Italy, Spain, Austria} from 60 single-trial classifications for each participant.

Figure 18. Significant structural similarities (partial orders of similarity differences invariant between the brain data and the linguistic data) for the set of words {Berlin, Rome, Warsaw, north, south, east, west, France, Greece, Poland} from 60 single-trial classifications for each participant.

Figure 19. Significant structural similarities (partial orders of similarity differences invariant between the brain data and the linguistic data) for the set of words {Madrid, Rome, Vienna, north, south, east, west, Spain, Italy, Austria} from 60 single-trial classifications for each participant.