LEMON: A Lightweight Facial Emotion Recognition System for Assistive Robotics Based on Dilated Residual Convolutional Neural Networks

Abstract

The development of a Social Intelligence System based on artificial intelligence is one of the cutting edge technologies in Assistive Robotics. Such systems need to create an empathic interaction with the users; therefore, it os required to include an Emotion Recognition (ER) framework which has to run, in near real-time, together with several other intelligent services. Most of the low-cost commercial robots, however, although more accessible by users and healthcare facilities, have to balance costs and effectiveness, resulting in under-performing hardware in terms of memory and processing unit. This aspect makes the design of the systems challenging, requiring a trade-off between the accuracy and the complexity of the adopted models. This paper proposes a compact and robust service for Assistive Robotics, called Lightweight EMotion recognitiON (LEMON), which uses image processing, Computer Vision and Deep Learning (DL) algorithms to recognize facial expressions. Specifically, the proposed DL model is based on Residual Convolutional Neural Networks with the combination of Dilated and Standard Convolution Layers. The first remarkable result is the few numbers (i.e., 1.6 Million) of parameters characterizing our model. In addition, Dilated Convolutions expand receptive fields exponentially with preserving resolution, less computation and memory cost to recognize the distinction among facial expressions by capturing the displacement of the pixels. Finally, to reduce the dying ReLU problem and improve the stability of the model, we apply an Exponential Linear Unit (ELU) activation function in the initial layers of the model. We have performed training and evaluation (via one- and five-fold cross validation) of the model with five datasets available in the community and one mixed dataset created by taking samples from all of them. With respect to the other approaches, our model achieves comparable results with a significant reduction in terms of the number of parameters.

Keywords: assistive robotics; computer vision; deep convolutional neural networks; emotion recognition; face recognition.

Figure 1

Residual Learning: a building block [27].

Figure 2

The LEMON Architecture pipeline and a representation of the residual bock including Convolution (left) and Identity Block (right).

Figure 3

Filter Configuration of LEMON Network. The red values represent the number of filters in the final layer of each block.

Figure 4

Our robotic platform.

Figure 5

Sample images from (a) CK+, (b) Jaffe, (c) KDEF, (d) Sase-FE and (e) TFEID Datasets.

Figure 6

Pipeline for Training and Evaluation: Including Preprocessing, Data Partition, Data Augmentation, Training and Evaluation.

Figure 7

Learning Curves (Loss) over the examined datasets.

Figure 8

Confusion matrices over the examined datasets. The shades of colors indicate the classification performance. Darker colors are associated with higher prediction accuracy.

Figure 9

The real-time results: (a) The entire distribution of the prediction time; (b) Illustrative examples of the detected emotions.