How the visual brain detects emotional changes in facial expressions: Evidence from driven and intrinsic brain oscillations

Abstract

The processing of facial expressions is often studied using static pictorial cues. Recent work, however, suggests that viewing changing expressions more robustly evokes physiological responses. Here, we examined the sensitivity of steady-state visual evoked potentials and intrinsic oscillatory brain activity to transient emotional changes in facial expressions. Twenty-two participants viewed sequences of grayscale faces periodically turned on and off at a rate of 17.5 Hz, to evoke flicker steady-state visual evoked potentials (ssVEPs) in visual cortex. Each sequence began with a neutral face (flickering for 2290 msec), immediately followed by a face from the same actor (also flickering for 2290 msec) with one of four expressions (happy, angry, fearful, or another neutral expression), followed by the initially presented neutral face (flickering for 1140 msec). The amplitude of the ssVEP and the power of intrinsic brain oscillations were analyzed, comparing the four expression-change conditions. We found a transient perturbation (reduction) of the ssVEP that was more pronounced after the neutral-to-angry change compared to the other conditions, at right posterior sensors. Induced alpha-band (8-13 Hz) power was reduced compared to baseline after each change. This reduction showed a central-occipital topography and was strongest in the subtlest and rarest neutral-to-neutral condition. Thus, the ssVEP indexed involvement of face-sensitive cortical areas in decoding affective expressions, whereas mid-occipital alpha power reduction reflected condition frequency rather than expression-specific processing, consistent with the role of alpha power changes in selective attention.

Keywords: Alpha-band oscillations; EEG; Face processing; Facial expressions; ssVEP.

Figure 1.

(A) Schematic of the experimental design. Each trial began with a neutral face presented on the screen for the initial 2.29 seconds (ONSET time window). Subsequently, the picture changed to a happy (H), angry (A), fearful (F), or a different neutral (N) expression of the same actor from 2.29 to 4.57 seconds (CHANGE time window). The trial ended with the initial neutral expression presented from 4.57 to 5.71 seconds (RETURN time window). The stimuli flickered at 17.5 Hz throughout the trial. A variable inter-trial interval (2–4 seconds) separated trials. (B) Example of happy, angry, fearful, and neutral stimuli are presented in a clockwise fashion.

Figure 2.

The 128-sensor HydroCel GSN recording montage. The highlighted gray sensors represent the selected mid-occipital cluster, comprising Oz and the six nearest neighbors. The highlighted black sensors represent the right occipito-temporal cluster, with PO8 and the six nearest neighbors.

Figure 3.

Grand mean time-locked averages of the voltages of the ssVEP over sensors Oz and PO8 across all the conditions and all the 22 participants. Facial expressions evoked pronounced 17.5 Hz oscillations, which are abruptly disrupted by the transient change of facial expressions within the trials. The beginning of each change period is represented by the vertical lines in the graphs. Note that the amplitude of the steady-state potential evoked in the representative PO8 right occipito-temporal sensor is higher compared to the Oz mid-occipital sensor.

Figure 4.

Frequency spectrum and topographical map of the steady-state visual evoked potential at the 17.5 Hz-driven frequency. Individually computed spectra were averaged across participants for each condition separately to obtain the grand mean frequency spectrum (neutral condition shown). Note the pronounced peak at the 17.5 Hz-driven frequency (left). The topographical map shows that the peak of the evoked 17.5 Hz response is focally localized around the occipital pole (right).

Figure 5.

Time-varying ssVEP amplitude averaged over seven occipital sensors for the angry, fearful, happy, and neutral facial expressions for all the participants (N=22) for the mid-occipital cluster and the right occipito-temporal cluster. Top: Hilbert waveforms show differences only for the right occipito-temporal cluster during the first change. The black line at the bottom of the top right plot represents the t-values for the comparison of the ssVEP amplitudes of angry minus neutral facial expressions at the right occipito-temporal cluster. The gray-shaded area shows the permutation-controlled t-threshold of −2.86. Bottom left: Topographical maps of the ssVEP differential amplitude (emotional minus neutral) are represented for the CHANGE time period, ranging from 2700 ms to 2900 ms. The ssVEP amplitude evoked after the change to angry faces were significantly reduced when compared to neutral dynamic faces. Bottom right: The bar plot illustrates the means during the first change window, with the ssVEP amplitude decreased more in response to angry faces compared to the other expressions, over the right occipito-temporal cluster. No significant effect was found for the mid-occipital cluster. * represents p < 0.05.

Figure 6.

Grand mean evolutionary spectra for sensor Oz, shown for each facial expression. Plots show time-varying amplitudes for frequencies varying from 1.94 Hz up to 22.54 Hz, and for the time range from −600 ms to 5800 ms relative to picture onset. White boxes indicate the time periods used for ANOVA (ONSET, CHANGE, and RETURN). Results are plotted as the percentage of the signal change, relative to pre-stimulus baseline. The white dashed lines represent 9.3 Hz, the center frequency of the wavelet showing differences in permutation controlled t-tests: Time periods that exceeded the critical t-threshold are illustrated by showing the t-values as black areas, comparing each emotional expression condition compared to the neutral condition. Note that significant differences between neutral and emotional expressions arise after the change and are sustained throughout the RETURN time window. An example topography of the alpha amplitude reduction following the RETURN from neutral to neutral is shown at the top right (4700 ms to 5100 ms, center frequencies, 9.30 to 12.24 Hz).

Figure 7.

Topographical distribution of t-values comparing alpha amplitude for the neutral-neutral condition with the remaining conditions, shown here for the wavelet centered at 9.3 Hz, and a time point in the RETURN window (5080 ms post-onset). Note that the maximum t-values are located near the midline, at parietal and occipital sensor locations, not at right-occipital locations.