Optimal processing of surface facial EMG to identify emotional expressions: A data-driven approach

Abstract

Surface facial electromyography (EMG) is commonly used to detect emotions from subtle facial expressions. Although there are established procedures for collecting EMG data and some aspects of their processing, there is little agreement among researchers about the optimal way to process the EMG signal, so that the study-unrelated variability (noise) is removed, and the emotion-related variability is best detected. The aim of the current paper was to establish an optimal processing pipeline for EMG data for identifying emotional expressions in facial muscles. We identified the most common processing steps from existing literature and created 72 processing pipelines that represented all the different processing choices. We applied these pipelines to a previously published dataset from a facial mimicry experiment, where 100 adult participants observed happy and sad facial expressions, whilst the activity of their facial muscles, zygomaticus major and corrugator supercilii, was recorded with EMG. We used a resampling approach and subsets of the original data to investigate the effect and robustness of different processing choices on the performance of a logistic regression model that predicted the mimicked emotion (happy/sad) from the EMG signal. In addition, we used a random forest model to identify the most important processing steps for the sensitivity of the logistic regression model. Three processing steps were found to be most impactful: baseline correction, standardisation within muscles, and standardisation within subjects. The chosen feature of interest and the signal averaging had little influence on the sensitivity to the effect. We recommend an optimal processing pipeline, share our code and data, and provide a step-by-step walkthrough for researchers.

Keywords: Emotion; Facial electromyography; Multiverse; Optimal pipeline; Surface electromyography.

Fig. 1

Schematic illustration of the study design and the positions of the electrodes assessing the activation over the ZM and CS facial muscles. Taken from Vacaru et al. (2021)

Fig. 2

A A diagram of processing steps and their possible sequences. All pipelines included a data averaging step, either during signal averaging (first step) or during data reduction (last step), but the data were never averaged twice. B An example pipeline, including (1) no signal averaging in the first step, (2) mean absolute value as a feature of interest, (3) division by baseline as a baseline correction, (4) z-scoring within each muscle within participants, (5) no z-scoring between muscles within participants, and (6) averaging across trials in the data reduction step. It corresponds to pipeline Aa_MAV_Bd_Ms_Sn (see Table 1 in Supplementary materials and Naming the processing pipelines)

Fig. 3

A The results of the analysis of the resampled data processed with different pipelines, with the logistic models predicting emotional expression (happy or sad). The area under the curve (AUC) represents the overall performance of the models, with higher AUC meaning better performance, and AUC > 0.5 indicating better performance than chance. The AUC is averaged over all 1500 subsamples of data, and standard deviation error bars are displayed for each pipeline. B The results for the top 24 performing pipelines (AUC > 0.75) are displayed.

Fig. 4

Random forest model variable importance, measured with mean decrease in accuracy, in predicting pipeline performance (measured with average AUC). The higher the variable importance, the more impact it had on the performance of the pipelines. Note: The signal averaging variable refers to the choice to average before or after other processing steps

Fig. 5

Partial dependence plot showing predicted pipeline AUC for each level of each variable in our random forest model. Higher expected AUC value indicates more positive impact on pipeline performance