Decoding Semantics from Dynamic Brain Activation Patterns: From Trials to Task in EEG/MEG Source Space

Abstract

The temporal dynamics within the semantic brain network and its dependence on stimulus and task parameters are still not well understood. Here, we addressed this by decoding task as well as stimulus information from source-estimated EEG/MEG human data. We presented the same visual word stimuli in a lexical decision (LD) and three semantic decision (SD) tasks. The meanings of the presented words varied across five semantic categories. Source space decoding was applied over time in five ROIs in the left hemisphere (anterior and posterior temporal lobe, inferior frontal gyrus, primary visual areas, and angular gyrus) and one in the right hemisphere (anterior temporal lobe). Task decoding produced sustained significant effects in all ROIs from 50 to 100 ms, both when categorizing tasks with different semantic demands (LD-SD) as well as for similar semantic tasks (SD-SD). In contrast, a semantic word category could only be decoded in lATL, rATL, PTC, and IFG, between 250 and 500 ms. Furthermore, we compared two approaches to source space decoding: conventional ROI-by-ROI decoding and combined-ROI decoding with back-projected activation patterns. The former produced more reliable results for word category decoding while the latter was more informative for task decoding. This indicates that task effects are distributed across the whole semantic network while stimulus effects are more focal. Our results demonstrate that the semantic network is widely distributed but that bilateral anterior temporal lobes together with control regions are particularly relevant for the processing of semantic information.

Keywords: MVPA; decoding; semantic cognition; semantics; task; visual word recognition.

Figure 1.

ROIs used in this study [based on Rahimi et al. (2022)].

Figure 2.

Evoked responses for several ROIs in source space for each task and SD tasks averaged together. The shadowed areas report standard error of the mean (LD, lexical decision; SD, semantic decision; “milk,” “fruit,” and “odor” decisions refer to single SD blocks).

Figure 3.

Task classification performance. For all plots, the shadowed area represents the standard error of the mean (across participants). A, Individual-ROI task classification accuracy (ROC AUC) when decoding LD from SD (left) and when decoding SD tasks (right). B, Root mean square of the activation patterns within each ROI, when decoding LD from SD (left) and when decoding SD tasks (right). C, Combined-ROI task classification accuracy (ROC AUC) when decoding LD from SD (left) and when decoding SD tasks (right).

Figure 4.

Single-word semantic category classification performance. For all plots, the shadowed area represents the standard error of the mean (across participants). The green color highlights the times when the cluster-based permutation correction test revealed a significant cluster (α = 0.05) where accuracy was above 0.5, and the red the clusters that are still significant after Bonferroni’s correction (across ROIs). A, Individual-ROI task classification accuracy (ROC AUC) when decoding semantic features in LD block. B, Individual-ROI task classification accuracy (ROC AUC) when decoding semantic features in SD blocks. C, Combined-ROI root mean square of the activation patterns within each ROI (left) and classification accuracy (ROC AUC; right), when decoding semantic features in LD block when decoding LD. D, Combined-ROI root mean square of the activation patterns within each ROI (left) and classification accuracy (ROC AUC; right), when decoding semantic features in SD block.

Figure 5.

Cross-task semantic category decodability. For all plots, the shadowed area represents the standard error of the mean (across participants). Time points highlighted in green are times when the cluster-based permutation correction test revealed a significant cluster (α = 0.05). In red, are clusters that are still significant after Bonferroni’s correction (across ROIs). A, Individual-ROI cross-decoding when the model was trained on SD trials and tested on LD trials. B, Individual-ROI cross-decoding when the model was trained on LD trials and tested on SD trials.

Figure 6.

Confusion Matrices for the decoding of semantic word categories. Each cell represents the temporally averaged probability of a predicted word category, given the true word category, separately for each of the significant temporal clusters found in the semantic category classification. Smaller values indicate lower probabilities. In all cells, abstract words were the most accurately predicted category, compared with the other (concrete) words.