The MPI facial expression database--a validated database of emotional and conversational facial expressions

Abstract

The ability to communicate is one of the core aspects of human life. For this, we use not only verbal but also nonverbal signals of remarkable complexity. Among the latter, facial expressions belong to the most important information channels. Despite the large variety of facial expressions we use in daily life, research on facial expressions has so far mostly focused on the emotional aspect. Consequently, most databases of facial expressions available to the research community also include only emotional expressions, neglecting the largely unexplored aspect of conversational expressions. To fill this gap, we present the MPI facial expression database, which contains a large variety of natural emotional and conversational expressions. The database contains 55 different facial expressions performed by 19 German participants. Expressions were elicited with the help of a method-acting protocol, which guarantees both well-defined and natural facial expressions. The method-acting protocol was based on every-day scenarios, which are used to define the necessary context information for each expression. All facial expressions are available in three repetitions, in two intensities, as well as from three different camera angles. A detailed frame annotation is provided, from which a dynamic and a static version of the database have been created. In addition to describing the database in detail, we also present the results of an experiment with two conditions that serve to validate the context scenarios as well as the naturalness and recognizability of the video sequences. Our results provide clear evidence that conversational expressions can be recognized surprisingly well from visual information alone. The MPI facial expression database will enable researchers from different research fields (including the perceptual and cognitive sciences, but also affective computing, as well as computer vision) to investigate the processing of a wider range of natural facial expressions.

Figure 1. Set-up of the video lab.

Figure shows the used set-up for expression recording. The expressions were recorded by three fully synchronized cameras. The models were sitting in front of the frontal camera and acted as if the central camera is a person to address. To facilitate this “face to face” scenario, the experimenter was standing behind the frontal camera.

Figure 2. Example of models.

Figure shows examples of four models out of the MPI facial expression database. Models were sitting in front of a black background wearing a black cloak. Moreover, they were wearing a black hat with six green markers on that worked as head tracking. The upper row shows the four models in a neutral position, whereas in the lower row models show a smile expression. Note that the expressions are available in a static and a dynamic version.

Figure 3. Context condition: Frequency of number of valid answers.

Frequency distribution of the number of expressions with a given number of valid answers for the context condition. Maximum number of valid answers is 10 as there were 10 participants.

Figure 4. Context condition: Frequency of participants' confidence.

Frequency distribution of participants' confidence ratings pooled over participants and expressions for the context condition. Confidence score 1 means “not confident at all” whereas 5 means “very confident”.

Figure 5. Visual condition: Frequency of valid answers.

Frequency distribution of the number of expressions with a given number of valid answers for the visual condition. Since the maximum number of valid answers for each expression is 100 (10 models * 10 participants), expressions were grouped together resulting in group increments of 10.

Figure 6. Visual condition: Mean number of valid answers per each actor.

Mean number of valid answers for each of the ten models sorted in descending order. Error bars present uncorrected confidence intervals.

Figure 7. Visual condition: Frequency of valid answers for all models for worst expressions.

Frequency distribution of the number of valid answers for each of the ten models for the expressions with the lowest number of valid answers for the visual condition. The abbreviations of the expressions are the following: arr = arrogant, bot = bothering, cont = contempt, dcar = don't care, paf = feeling pain, smsad = smiling nostalgic, smyeah = smiling “Yeah right!”, trdoof = doe eyed.

Figure 8. Visual condition: Frequency of valid answers for all models for best expressions.

Frequency distribution of the number of valid answers for each of the ten models for the expressions with the highest number of valid answers for the visual condition. The abbreviations of the expressions are the following: disrel = reluctant disagreeing, discon = considered disagreeing, ncon = not convinced, disag = disagreeing, reco = thinking considering, agcons = considered agreeing.

Figure 9. Visual condition: Frequency of naturalness scores for each expression stimulus.

Frequency distribution of participants' naturalness ratings pooled over participants and expressions for the visual condition. Naturalness score 1 means “extremely posed facial expression” whereas 5 means “natural expression as it would occur during a conversation”.

Figure 10. Visual condition: Average naturalness scores for each model and their corresponding confidence intervals.

Mean naturalness scores for each model and their corresponding confidence intervals for the visual condition. Grey horizontal line indicates the mean naturalness ratings over all models.

Figure 11. Results of the naming task for the conversational expressions in the two conditions.

Plot presenting the mean amount of invalid answers for both conditions for each of the conversational facial expression. The abbreviations of the expressions are the following: agcons = considered agreeing, agcont = agree and continue, agr = agree, agrel = reluctant agreeing, aha = lightbulb moment, bor = bored, bot = bothering, conf = confused, dcar = don't care, dhear = don't hear, disag = disagreeing, dis = disbelieve, discon = considered disagreeing, disrel = reluctant disagreeing, dkno = don't know, dund = don't understand, imneg = imagine negative, impos = imagine positive, impr = impressed, ins = insecure, mitl = compassion, ncon = not convinced, re = thinking/remembering, reco = thinking considering, reneg = thinking negative, repos = thinking positive, reps = problem solving, tir = tired.