Development of a Universal Validation Protocol and an Open-Source Database for Multi-Contextual Facial Expression Recognition

Abstract

Facial expression recognition (FER) poses a complex challenge due to diverse factors such as facial morphology variations, lighting conditions, and cultural nuances in emotion representation. To address these hurdles, specific FER algorithms leverage advanced data analysis for inferring emotional states from facial expressions. In this study, we introduce a universal validation methodology assessing any FER algorithm's performance through a web application where subjects respond to emotive images. We present the labelled data database, FeelPix, generated from facial landmark coordinates during FER algorithm validation. FeelPix is available to train and test generic FER algorithms, accurately identifying users' facial expressions. A testing algorithm classifies emotions based on FeelPix data, ensuring its reliability. Designed as a computationally lightweight solution, it finds applications in online systems. Our contribution improves facial expression recognition, enabling the identification and interpretation of emotions associated with facial expressions, offering profound insights into individuals' emotional reactions. This contribution has implications for healthcare, security, human-computer interaction, and entertainment.

Keywords: affective computing; facial expression recognition; facial landmarks; labelled data database; machine-learning algorithms.

Figure 1

Age distribution among individuals involved in the experimental validation process of the Facial Expression Recognition algorithm under investigation.

Figure 2

Examples of the graphical interface pages designed for the experimental protocol: (a) example of the starting page; (b) example of the selection page. (Translated from the original Italian version).

Figure 3

Facial landmarks coordinates elaboration process: (a) coordinates of the landmarks as obtained by the algorithm under investigation; (b) coordinates after applying normalization; (c) coordinates of the 22 key points selected for their high degree of informativeness.

Figure 4

Validation metrics—Precision, F-score, and G-mean—pertaining to the investigated algorithm during the validation process. These metrics are presented for each discrete emotion category, including neutrality.

Figure 5

Validation metrics acquired from the algorithm developed to ascertain the integrity of the proposed FeelPix database. These metrics include precision, F-score, and G-mean, and are displayed for each emotion category.

Figure 6

Comparing the outcomes produced by the algorithm under investigation during the validation process with those achieved by the algorithm developed for validating the proposed database. The visualization includes precision, F-score, and G-mean metrics, shedding light on the performance of each algorithm across diverse emotion categories.