Decoding Neural Representations of Affective Scenes in Retinotopic Visual Cortex

Abstract

The perception of opportunities and threats in complex visual scenes represents one of the main functions of the human visual system. The underlying neurophysiology is often studied by having observers view pictures varying in affective content. It has been shown that viewing emotionally engaging, compared with neutral, pictures (1) heightens blood flow in limbic, frontoparietal, and anterior visual structures and (2) enhances the late positive event-related potential (LPP). The role of retinotopic visual cortex in this process has, however, been contentious, with competing theories predicting the presence versus absence of emotion-specific signals in retinotopic visual areas. Recording simultaneous electroencephalography-functional magnetic resonance imaging while observers viewed pleasant, unpleasant, and neutral affective pictures, and applying multivariate pattern analysis, we found that (1) unpleasant versus neutral and pleasant versus neutral decoding accuracy were well above chance level in retinotopic visual areas, (2) decoding accuracy in ventral visual cortex (VVC), but not in early or dorsal visual cortex, was correlated with LPP, and (3) effective connectivity from amygdala to VVC predicted unpleasant versus neutral decoding accuracy, whereas effective connectivity from ventral frontal cortex to VVC predicted pleasant versus neutral decoding accuracy. These results suggest that affective scenes evoke valence-specific neural representations in retinotopic visual cortex and that these representations are influenced by reentry signals from anterior brain regions.

Keywords: affective scenes; amygdala; late positive potential; multivariate pattern analysis; visual cortex.

Figure 1

Picture viewing paradigm. There were five sessions. Each session lasted 7 min. A total of 60 IAPS pictures, including 20 pleasant, 20 unpleasant, and 20 neutral, were presented in each session, the order of which randomly varied from session to session. Each picture lasted 3 s and was followed by a fixation period referred to as intertrial interval (ITI) (2.8 or 4.3 s). Participants were required to fixate on the cross in the center of screen throughout the session.

Figure 2

Univariate fMRI and ERP analysis. (A) Activation map (P < 0.05, FDR) contrasting unpleasant versus neutral pictures. (B) Activation map (P < 0.05, FDR) contrasting pleasant versus neutral pictures. (C) Grand average ERP (n = 20) at Pz showing ERP evoked by three classes of pictures (left) and scalp topography of LPP enhancement (300–800 ms after picture onset). PPC, posterior parietal cortex; OFC, orbital frontal cortex; OTJ, occipitotemporal junction.

Figure 3

MVPA decoding analysis of neural representations of emotional scenes in retinotopic visual cortex. (A) Retinotopic ROIs visualized on the flattened brain. (B) Group average decoding accuracy between unpleasant versus neutral and pleasant versus neutral in different ROIs. Dashed line indicates the statistical significance threshold (54%). (C) Comparison between visual cortical contribution to the representation of affective scenes revealed by (top) univariate activation analysis (data from Fig. 2A,B replotted here) and by (bottom) multivariate decoding from B.

Figure 4

MVPA decoding of pleasant versus unpleasant scenes. (A) Normative valence and arousal ratings of unpleasant and pleasant images used in this study. The error bar depicts the standard error of the mean for the normative ratings of 20 pictures in one category. Arousal is not significantly different between the two classes of pictures, whereas pleasant pictures have significantly higher valence than unpleasant pictures. (B) Group average decoding accuracy between unpleasant and pleasant in retinotopic ROIs. Dashed line indicates the statistical significance threshold (54%) at P < 0.001 according to a random permutationtest.

Figure 5

Relation between decoding accuracy and measures of signal reentry. (A) Anatomical location of EVC, VVC, and DVC on flattened brain. (B) LPP – decoding accuracy correlation for unpleasant versus neutral (left) and pleasant versus neutral (right) in EVC, VVC, and DVC. Only decoding accuracy in VVC is significantly correlated with LPP. (C) Scatter plots showing relationship between LPP and decoding accuracy for unpleasant versus neutral (left) and pleasant versus neutral (right) in VVC. (D) Relationship between amygdala→VVC effective connectivity (EC) and decoding accuracy in VVC. This relationship is only significant in unpleasant versus neutral decoding (^*P < 0.05).

Figure 6

VVC-seeded whole-brain effective connectivity analysis. (A) Brain maps showing voxels whose effective connectivity into VVC predicts VVC pleasant vs neutral decoding accuracy. (B) Measured VVC pleasant vs neutral decoding accuracy vs predicted VVC pleasant vs neutral decoding accuracy according to a linear model accounting for the collective contributions of reentry signaling from regions identified in panel A (see Results). (C) Brain maps showing voxels whose effective connectivity into VVC predicts VVC unpleasant vs neutral decoding accuracy. (D) Measured VVC unpleasant vs neutral decoding accuracy vs predicted VVC unpleasant vs neutral decoding accuracy according to a linear model accounting for the collective contributions of reentry signaling from regions identified in panel C (see Results). STG, superior temporal gyrus; STS, superior temporal sulcus; IFG, inferior frontal gyrus; VLPFC, ventral lateral prefrontal cortex. All maps were thresholded at R > 0.61, p < 0.005 and clusters containing more than 10 contiguous such voxels are shown.

Figure 7

Effects of picture repetition. (A) LPP enhancement as a function of run. (B) Linear fits to LPP as a function of run for each individual participant (n = 20). The slopes were not significantly different from zero (P = 0.51 for pleasant vs. neutral and P = 0.17 for unpleasant vs. neutral). (C) Decoding accuracy as a function of run. (D) Linear fits to decoding accuracy as a function of run for each individual participant (n = 20). The slopes were not significantly different from zero (P = 0.16 for pleasant vs. neutral and P = 0.07 for unpleasant vs. neutral).