Training Emotion Recognition Accuracy: Results for Multimodal Expressions and Facial Micro Expressions

Abstract

Nonverbal emotion recognition accuracy (ERA) is a central feature of successful communication and interaction, and is of importance for many professions. We developed and evaluated two ERA training programs-one focusing on dynamic multimodal expressions (audio, video, audio-video) and one focusing on facial micro expressions. Sixty-seven subjects were randomized to one of two experimental groups (multimodal, micro expression) or an active control group (emotional working memory task). Participants trained once weekly with a brief computerized training program for three consecutive weeks. Pre-post outcome measures consisted of a multimodal ERA task, a micro expression recognition task, and a task about patients' emotional cues. Post measurement took place approximately a week after the last training session. Non-parametric mixed analyses of variance using the Aligned Rank Transform were used to evaluate the effectiveness of the training programs. Results showed that multimodal training was significantly more effective in improving multimodal ERA compared to micro expression training or the control training; and the micro expression training was significantly more effective in improving micro expression ERA compared to the other two training conditions. Both pre-post effects can be interpreted as large. No group differences were found for the outcome measure about recognizing patients' emotion cues. There were no transfer effects of the training programs, meaning that participants only improved significantly for the specific facet of ERA that they had trained on. Further, low baseline ERA was associated with larger ERA improvements. Results are discussed with regard to methodological and conceptual aspects, and practical implications and future directions are explored.

Keywords: emotion recognition; emotion recognition training; micro expression recognition; multimodal emotion recognition; nonverbal communication.

Figure 1

Violin plot with box plots of the training effects for the three main ERA measure: (A) ERAM total score, (B) MICRO score, (C) PECT score. The violin plots display the kernel probability density of the data at the different values for the three training groups. The box plots within the violin plots include the median (–) and the interquartile range (box), as well as the minimum and maximum (whiskers). Means were added in form of rhombuses. Small black dots display outliers.