Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May 28;24(11):3484.
doi: 10.3390/s24113484.

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

Rafael Pereira  1 , Carla Mendes  1 , José Ribeiro  1 , Roberto Ribeiro  1 , Rolando Miragaia  1 , Nuno Rodrigues  1 , Nuno Costa  1 , António Pereira  1   2
Affiliations

Systematic Review of Emotion Detection with Computer Vision and Deep Learning

Rafael Pereira et al. Sensors (Basel). .

Abstract

Emotion recognition has become increasingly important in the field of Deep Learning (DL) and computer vision due to its broad applicability by using human-computer interaction (HCI) in areas such as psychology, healthcare, and entertainment. In this paper, we conduct a systematic review of facial and pose emotion recognition using DL and computer vision, analyzing and evaluating 77 papers from different sources under Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our review covers several topics, including the scope and purpose of the studies, the methods employed, and the used datasets. The scope of this work is to conduct a systematic review of facial and pose emotion recognition using DL methods and computer vision. The studies were categorized based on a proposed taxonomy that describes the type of expressions used for emotion detection, the testing environment, the currently relevant DL methods, and the datasets used. The taxonomy of methods in our review includes Convolutional Neural Network (CNN), Faster Region-based Convolutional Neural Network (R-CNN), Vision Transformer (ViT), and "Other NNs", which are the most commonly used models in the analyzed studies, indicating their trendiness in the field. Hybrid and augmented models are not explicitly categorized within this taxonomy, but they are still important to the field. This review offers an understanding of state-of-the-art computer vision algorithms and datasets for emotion recognition through facial expressions and body poses, allowing researchers to understand its fundamental components and trends.

Keywords: computer vision; deep learning; emotion detection; emotion recognition; systematic review.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflicts of interest.

Figures

Figure 1
Figure 1
Systematic review process.
Figure 2
Figure 2
Study selection process.
Figure 3
Figure 3
Results taxonomy.
Figure 4
Figure 4
Number of studies employing performance improvement techniques. The graph compares the number of studies that mentioned the use of fine-tuning, hyper-parameter tuning, and batch normalization, out of a total of 77 studies fully analyzed.
See this image and copyright information in PMC

References

    1. Lecun Y., Bengio Y., Hinton G. Deep Learning. Nature. 2015;521:436–444. doi: 10.1038/nature14539. - DOI - PubMed
    1. Goodfellow I., Bengio Y., Courville A. Deep Learning. MIT Press; Cambridge, MA, USA: 2016.
    1. Chollet F. Deep Learning with Python. Manning Publications; Shelter Island, NY, USA: 2018.
    1. Pereira R., Mendes C., Ribeiro R., Ribeiro J., Pereira A. International Symposium on Ambient Intelligence. Volume 770 LNNS. Springer; Cham, Switzerland: 2023. Human-in-the-loop AAL Approach to Emotion Capture and Classification; pp. 123–132. Lecture Notes in Networks and Systems. - DOI
    1. Mendes C., Pereira R., Ribeiro J., Rodrigues N., Pereira A. International Symposium on Ambient Intelligence. Volume 770 LNNS. Springer; Cham, Switzerland: 2023. Chatto: An Emotionally Intelligent Avatar for Elderly Care in Ambient Assisted Living; pp. 93–102. Lecture Notes in Networks and Systems. - DOI

Publication types