Audio-Visual Interactions During Emotion Processing in Bicultural Bilinguals

Abstract

Despite the growing number of bicultural bilinguals in the world, the way in which multisensory emotions are evaluated by bilinguals who identify with two or more cultures remains unknown. In the present study, Chinese-English bicultural bilinguals from Singapore viewed Asian or Caucasian faces and heard Mandarin or English speech, and evaluated the emotion from one of the two simultaneously-presented modalities. Reliance on the visual modality was greater when bicultural bilinguals processed Western audio-visual emotion information. Although no differences between modalities emerged when processing East-Asian audio-visual emotion information, correlations revealed that bicultural bilinguals increased their reliance on the auditory modality with more daily exposure to East-Asian cultures. Greater interference from the irrelevant modality was observed for Asian faces paired with English speech than for Caucasian faces paired with Mandarin speech. We conclude that processing of emotion in bicultural bilinguals is guided by culture-specific norms, and that familiarity influences how the emotions of those who speak a foreign language are perceived and evaluated.

Keywords: bicultural bilinguals; biculturalism; cultural frame switching; emotion; emotion perception; modality dominance.

Figure 1.

Examples of bimodal stimuli by task (voice vs. face), culture (East vs. West), and match (match vs. mismatch). The green box denotes the target modality that participants were instructed to respond to in each condition. For both match and mismatch conditions, trials that required participants to judge the emotion from the Asian face (face task) or Mandarin speech (voice task) were labeled as East, whereas trials that required participants to judge the emotion from the Caucasian face (face task) or English speech (voice task) were labeled as West. (Asian face—Image ID ang118: Adapted with permission from Chen and Yen (2007); Mandarin pseudo-sentence: Adapted with permission from Liu and Pell (2012); Caucasian face—Image ID AM08ANS: Adapted with permission from Lundqvist et al. (1998); English pseudo-sentence: Adapted with permission from Pell et al. (2009)).

Figure 2.

The effect of culture (East vs. West) and task (face vs. voice) on modality dominance in accuracy rates. Modality dominance was calculated by subtracting the raw accuracy score of the incongruent condition from the congruent condition. A larger modality dominance in the face task reflects greater interference from the voice, whereas a larger modality dominance in the voice task reflects greater interference from the face. The voice task (grey) produced a larger modality dominance than the face task (white) for audio-visual emotional information from the West (i.e., visual dominance). No difference in modality dominance between tasks emerged for emotional audio-visual information from the East. *p < .05.

Figure 3.

The interaction of culture (East vs. West), task (face vs. voice), and match (match vs. mismatch) on modality dominance in intensity ratings. Modality dominance was calculated by subtracting the intensity ratings of the incongruent condition from the congruent condition. A larger modality dominance in the face task reflects greater interference from the voice, while a larger modality dominance in the voice task reflects greater interference from the face. When both the auditory and visual inputs were from the same culture, the voice task produced a larger modality dominance than the face task. In the cultural mismatch conditions, where the auditory and visual inputs were from different cultures, the modality dominance changed depending on the target’s culture. When the target’s culture was Eastern, the face task (i.e., Asian face) produced a larger modality dominance than the voice task (i.e., Mandarin speech). In contrast, when the target’s culture was Western, the voice task (i.e., English speech) produced a larger modality dominance than the face task (i.e., Caucasian face). *p < .05.

Figure 4.

Scatter plots of the correlations between auditory dominance and exposure to East-Asian cultures when both the auditory and visual inputs were from the East (A), and correlations between visual dominance and exposure to Western cultures when both the auditory and visual inputs were from the West (B). The shaded gray band around the regression line (in blue) is the 95% confidence interval.