Real-time emotion detection by quantitative facial motion analysis

Abstract

Background: Research into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools.

Methods: To address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis.

Results: We identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias.

Limitations: Our sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals.

Conclusions: We demonstrate that DISC-based facial analysis can be used to reliably identify an individual's emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.

Fig 1. Heatmaps derived from the results of DISC analysis of representative frames for each emotion in a single participant.

The top three panels are the original images, whereas the bottom three are the same images with superimposed heatmaps showing magnitude of movement from the baseline (neutral) frames. Units are in pixels.

Fig 2. Heatmaps showing magnitude of facial movement in response to happy and sad images for each participant.

Heatmaps were generated from the averaged DISC–calculated displacement across all happy and sad frames for that individual. Numbers represent each participant in the study. Participant 1 declined to have their face included in the publication of this data.

Fig 3. Average magnitude of facial movement in response to happy and sad images across all participants.

Units are in pixels.

Fig 4. Similarity matrices of DISC results from each frame of each participant’s video.

Matrices are numbered 1–10, corresponding to each participant’s ID. The matrix for Participant 1 is enlarged to show matrix organization. A value of 1.0 signifies 100% similarity between two frames and a value of 0 signifies the absence of any similarity.

Fig 5. Plot of first and second principal components of DISC–processed displacement across all participants.

Red squares represent frames from the happy image–viewing period, whereas blue diamonds represent frames from the sad image–viewing period. Black circles signify neutral frames. Large shapes indicate the averages for each participant. Gray lines serve to connect the averages for happy and sad for individual participants.

Fig 6

Temporal changes of average facial movement for each individual (ghosted lines) and across all participants (prominent line) over the duration of (A) happy and (B) sad image presentation. Dashed vertical lines represent the presentation of a new image.

Fig 7. Confusion matrices of the two methods.

(A)–(C): Different classifiers using the DISC method. (D): Amazon Rekognition. On the y–axis are the true emotion labels for the images; on the x–axis are the predicted emotion labels. The numbers in the plots indicate percentages of predicted labels for each true label. Correct predictions are along the diagonal of the matrix, as they are indicated by the boxes at the intersection of the same true and predicted labels on each axis. The Amazon Rekognition software contains seven emotion labels by default and this cannot be modified by the user. Abbreviations: DISC (digital image speckle correlation), SLR (sparse logistic regression), MLP (multi–layer perceptron), CNN (convolutional neural network).