Facial expression recognition (FER) survey: a vision, architectural elements, and future directions

Abstract

With the cutting-edge advancements in computer vision, facial expression recognition (FER) is an active research area due to its broad practical applications. It has been utilized in various fields, including education, advertising and marketing, entertainment and gaming, health, and transportation. The facial expression recognition-based systems are rapidly evolving due to new challenges, and significant research studies have been conducted on both basic and compound facial expressions of emotions; however, measuring emotions is challenging. Fueled by the recent advancements and challenges to the FER systems, in this article, we have discussed the basics of FER and architectural elements, FER applications and use-cases, FER-based global leading companies, interconnection between FER, Internet of Things (IoT) and Cloud computing, summarize open challenges in-depth to FER technologies, and future directions through utilizing Preferred Reporting Items for Systematic reviews and Meta Analyses Method (PRISMA). In the end, the conclusion and future thoughts are discussed. By overcoming the identified challenges and future directions in this research study, researchers will revolutionize the discipline of facial expression recognition in the future.

Keywords: Basic & compound emotions; Cloud computing; Computer vision; Emotion recognition technology; Internet of Things (IoT).

Figure 1. PRISMA review protocol.

Figure 2. Architecture elements of FER system.

Image source credit: Girl, Image by brgfx on Freepik (https://www.freepik.com/).

Figure 3. Applications of facial expression recognition.

Image source credits: Doctors and nurses working at the hospital, Image by brgfx on Freepik; Digital marketing doodles, Image by Freepik; Collection of hand-drawn video game, Image by Freepik; Different kinds of transportations, Image by brgfx on Freepik; A plain sketch of a teacher, Image by brgfx on Freepik (https://www.freepik.com/).

Figure 4. Smart city framework.

Image source credits: Smart home, Image by Freepik; Boy in yellow shirt, Image by brgfx on Freepik (https://www.freepik.com/). CCTV, https://www.rawpixel.com/image/6755502/png-sticker-public-domain, CC0; Bluetooth, Hospital, https://clipart-library.com/clipart/36700.htm, Non commercial use only; Devices on marble table, Image by Freepik; A sticker template with a laptop isolated, Image by brgfx on Freepik; Wifi Icon Vector, Image by iyikon on vecteezy.com; Wifi icon, Image by seetwo on vecteezy.com; Cloud computing concept, Image by macrovector on Freepik; Learning Deep Algorithm Data, Image by flatart on vecteezy.com; Girl in science gown writing note on white, Image by brgfx on Freepik.