Can deepfakes be used to study emotion perception? A comparison of dynamic face stimuli

Abstract

Video recordings accurately capture facial expression movements; however, they are difficult for face perception researchers to standardise and manipulate. For this reason, dynamic morphs of photographs are often used, despite their lack of naturalistic facial motion. This study aimed to investigate how humans perceive emotions from faces using real videos and two different approaches to artificially generating dynamic expressions - dynamic morphs, and AI-synthesised deepfakes. Our participants perceived dynamic morphed expressions as less intense when compared with videos (all emotions) and deepfakes (fearful, happy, sad). Videos and deepfakes were perceived similarly. Additionally, they perceived morphed happiness and sadness, but not morphed anger or fear, as less genuine than other formats. Our findings support previous research indicating that social responses to morphed emotions are not representative of those to video recordings. The findings also suggest that deepfakes may offer a more suitable standardized stimulus type compared to morphs. Additionally, qualitative data were collected from participants and analysed using ChatGPT, a large language model. ChatGPT successfully identified themes in the data consistent with those identified by an independent human researcher. According to this analysis, our participants perceived dynamic morphs as less natural compared with videos and deepfakes. That participants perceived deepfakes and videos similarly suggests that deepfakes effectively replicate natural facial movements, making them a promising alternative for face perception research. The study contributes to the growing body of research exploring the usefulness of generative artificial intelligence for advancing the study of human perception.

Keywords: Deepfake; Dynamic faces; Emotion; Face perception; Generative AI.

Fig. 1

Mask placement on deepfake (DF) types. Note. Shaded areas indicate the area that will be replaced with another identity. Face-swaps are the most common form of deepfakes, and require advanced mask blending techniques. Whole-head and puppetry deepfakes require all training images to portray the same hairstyle. Puppetry deepfakes replace all visible features with another identity and are the only deepfake type that does not require similar skin tones

Fig. 2

Example dynamic morph landmark placement on one actor. Note. Correctly placed landmarks at corresponding points on the neutral (left) and peak (right) results in a clear transition without overlap. The figure shows the resulting video frame (lower) that would be seen at the midpoint of the morphed video sequence

Fig. 3

Example source and destination that contribute to a deepfake. Note. The final deepfake blends the identity of one video (the source) with the temporal dynamics of another video (the destination). In puppetry deepfakes, the identity in the destination remains unseen in the final deepfake. If the destination blinks at the 5th frame of an expression transition, the source identity will also blink in the 5th frame of the final deepfake

Fig. 4

Trial sequence: 1200-ms stimulus followed by strength and genuineness questions. Note. The stimulus sequence included a 200-ms neutral freeze-frame, 800-ms transition, and then a 200-ms peak freeze-frame. Participants then selected a response for questions related to expression strength and then emotion genuineness

Fig. 5

Strength ratings of expressions for each display type. Note. Distribution and boxplots of mean strength ratings (1–5 Likert scale) per emotion for videos (V; green), morphs (M; orange), and deepfakes (D; purple); *p_c < .05, **p_c < .01. Higher scores indicate higher perceived strength of the expression

Fig. 6

Genuineness ratings of emotions for videos (V), morphs (M), and deepfakes (D). Note. Distribution and boxplots of mean ratings (1–5 Likert scale) per emotion for Videos (V; green), morphs (M; orange), and Deepfakes (D; purple); **p < .01. Higher scores indicate higher perceived emotion genuineness

Fig. 7

Hierarchy chart of codes for videos, deepfakes, and dynamic morphs. Note. Themes for each display type were coded using NVIVO qualitative data analysis software. Larger boxes indicate larger amounts