Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Sep 21;20(18):5391.
doi: 10.3390/s20185391.

Development of a Robust Multi-Scale Featured Local Binary Pattern for Improved Facial Expression Recognition

Suraiya Yasmin  1 , Refat Khan Pathan  2 , Munmun Biswas  2 , Mayeen Uddin Khandaker  3 , Mohammad Rashed Iqbal Faruque  4
Affiliations

Development of a Robust Multi-Scale Featured Local Binary Pattern for Improved Facial Expression Recognition

Suraiya Yasmin et al. Sensors (Basel). .

Abstract

Compelling facial expression recognition (FER) processes have been utilized in very successful fields like computer vision, robotics, artificial intelligence, and dynamic texture recognition. However, the FER's critical problem with traditional local binary pattern (LBP) is the loss of neighboring pixels related to different scales that can affect the texture of facial images. To overcome such limitations, this study describes a new extended LBP method to extract feature vectors from images, detecting each image from facial expressions. The proposed method is based on the bitwise AND operation of two rotational kernels applied on LBP(8,1) and LBP(8,2) and utilizes two accessible datasets. Firstly, the facial parts are detected and the essential components of a face are observed, such as eyes, nose, and lips. The portion of the face is then cropped to reduce the dimensions and an unsharp masking kernel is applied to sharpen the image. The filtered images then go through the feature extraction method and wait for the classification process. Four machine learning classifiers were used to verify the proposed method. This study shows that the proposed multi-scale featured local binary pattern (MSFLBP), together with Support Vector Machine (SVM), outperformed the recent LBP-based state-of-the-art approaches resulting in an accuracy of 99.12% for the Extended Cohn-Kanade (CK+) dataset and 89.08% for the Karolinska Directed Emotional Faces (KDEF) dataset.

Keywords: computer vision; facial expression recognition system; machine learning; multi-scale featured local binary pattern; unsharp masking.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Sample face image from Extended Cohn–Kanade (CK+) and Karolinska Directed Emotional Faces (KDEF) datasets.
Figure 2
Figure 2
Pre-processing steps.
Figure 3
Figure 3
Unsharp masking kernel.
Figure 4
Figure 4
Feature extraction process.
Figure 5
Figure 5
Feature extraction algorithm.
Figure 6
Figure 6
(a) Sample image segment of 3 × 3, (b) Description of Local Binary Pattern (LBP) (8, 1): kernel value.
Figure 7
Figure 7
Calculation of LBP (8, 1).
Figure 8
Figure 8
Description of LBP (8, 2): kernel value.
Figure 9
Figure 9
Calculation of LBP (8, 2).
Figure 10
Figure 10
Converting process of selected geographical features of a histogram.
Figure 11
Figure 11
(a) Regular data, (b) Normalized data. Axis values are two feature values before (a) and after (b) normalization.
Figure 12
Figure 12
KDEF (KNN: 38.09, QA: 64.52, Tree: 68.33, SVM: 89.05), CK+ (KNN: 83.05, QA: 88.70, Tree: 96.01, SVM: 99.12).
Figure 13
Figure 13
Dataset: CK+, Precision, Recall and F1 score shown for SVM.
Figure 14
Figure 14
Dataset: KDEF, Precision, Recall, and F1 score is shown for SVM.
See this image and copyright information in PMC

References

    1. Yu Z., Zhang C. Image based static facial expression recognition with multiple deep network learning; Proceedings of the ICMI 2015-Proceedings of the 2015 ACM International Conference on Multimodal Interaction; Washington, DC, USA. 9–13 November 2015.
    1. Kahou S.E., Michalski V., Konda K., Memisevic R., Pal C. Recurrent neural networks for emotion recognition in video; Proceedings of the ICMI 2015-Proceedings of the 2015 ACM International Conference on Multimodal Interaction; Washington, DC, USA. 9–13 November 2015.
    1. Liu M., Li S., Shan S., Wang R., Chen X. Asian Conference on Computer Vision, Proceedings of the Lecture Notes in Computer Science. Springer International Publishing; Cham, Switzerland: 2015. Deeply learning deformable facial action parts model for dynamic expression analysis.
    1. Shan C., Braspenning R. Recognizing Facial Expressions Automatically from Video. In: Nakashima H., Aghajan H., Augusto J.C., editors. Handbook of Ambient Intelligence and Smart Environments. Springer International Publishing; Cham, Switzerland: 2010.
    1. Yang B., Cao J., Ni R., Zhang Y. Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images. IEEE Access. 2017 doi: 10.1109/ACCESS.2017.2784096. - DOI