Mask exposure during COVID-19 changes emotional face processing

Abstract

Faces are one of the key ways that we obtain social information about others. They allow people to identify individuals, understand conversational cues, and make judgements about others' mental states. When the COVID-19 pandemic hit the United States, widespread mask-wearing practices were implemented, causing a shift in the way Americans typically interact. This introduction of masks into social exchanges posed a potential challenge-how would people make these important inferences about others when a large source of information was no longer available? We conducted two studies that investigated the impact of mask exposure on emotion perception. In particular, we measured how participants used facial landmarks (visual cues) and the expressed valence and arousal (affective cues), to make similarity judgements about pairs of emotion faces. Study 1 found that in August 2020, participants with higher levels of mask exposure used cues from the eyes to a greater extent when judging emotion similarity than participants with less mask exposure. Study 2 measured participants' emotion perception in both April and September 2020 -before and after widespread mask adoption-in the same group of participants to examine changes in the use of facial cues over time. Results revealed an overall increase in the use of visual cues from April to September. Further, as mask exposure increased, people with the most social interaction showed the largest increase in the use of visual facial cues. These results provide evidence that a shift has occurred in how people process faces such that the more people are interacting with others that are wearing masks, the more they have learned to focus on visual cues from the eye area of the face.

Fig 1. Example trials for face similarity task.

Each trial presents a pair of emotional faces. Participants rate the similarity of the expressed emotions. The images pictured here are similar, but not identical, to the original images used in the study, and are for illustrative purposes only.

Fig 2. Ground truth calculations.

A) The ground truth affective similarity is based on valence and arousal ratings. These ratings served as x-y coordinates for each face. The affective similarity score was calculated as the Euclidean distance between these values. B) The ground truth visual similarity is based on the Euclidean distances between all 68 facial landmarks. These landmarks were obtained from automated software (ResNext) for each face. Distances between each landmark were represented in a 68x68 matrix for each image. The global visual similarity score was calculated as the correlation between these two vectorized matrices.

Fig 3. Cue use calculations.

The participants’ similarity ratings were predicted by A) the affective similarity score, to calculate the affective cue use score, and B) the visual similarity score to calculate the global visual cue use score.

Fig 4. Effect of mask exposure on eye cue use in Study 1.

Use of eye cues in emotion judgements increased as exposure to masks increased. Transparent band reflects 95% confidence interval. Created using ggplot2 (v3.3.0) in RStudio.

Fig 5. Change in facial cue use between time 1 and time 2 in Study 2.

A) Eye and mouth cues use increased from time 1 to time 2. B) Arousal cue use decreased from time 1 to time 2. * p < 0.001. Error bars represent 95% confidence intervals. Created using the seaborn package (v0.10.0) for Python (v3.7.6).

Fig 6. Effect of mask exposure on global visual cue use as a function of social interaction in Study 2.

Social interaction mediated the relationship between mask exposure and visual cue use between time 1 and time 2. People with the least amount of social interaction had the greatest decrease in global visual cue use, and people with the most amount of social interaction had the greatest increase in use of global visual cues. Transparent bands reflect 95% confidence intervals.