Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines

Abstract

Facial expressions are widely used in the behavioral interpretation of emotions, cognitive science, and social interactions. In this paper, we present a novel method for fully automatic facial expression recognition in facial image sequences. As the facial expression evolves over time facial landmarks are automatically tracked in consecutive video frames, using displacements based on elastic bunch graph matching displacement estimation. Feature vectors from individual landmarks, as well as pairs of landmarks tracking results are extracted, and normalized, with respect to the first frame in the sequence. The prototypical expression sequence for each class of facial expression is formed, by taking the median of the landmark tracking results from the training facial expression sequences. Multi-class AdaBoost with dynamic time warping similarity distance between the feature vector of input facial expression and prototypical facial expression, is used as a weak classifier to select the subset of discriminative feature vectors. Finally, two methods for facial expression recognition are presented, either by using multi-class AdaBoost with dynamic time warping, or by using support vector machine on the boosted feature vectors. The results on the Cohn-Kanade (CK+) facial expression database show a recognition accuracy of 95.17% and 97.35% using multi-class AdaBoost and support vector machines, respectively.

Figure 1.

Overall block diagram of the landmark initialization and tracking process.

Figure 2.

Examples of the result of facial landmark tracking.

Figure 3.

Example of landmark tracking sequences before (First row) and after (Second row) normalization.

Figure 4.

Maximum intensity frame of each facial expression prototype.

Figure 5.

The first few (20, 40, 60, 80, and 100) features selected using multi-class AdaBoost. A blue dot indicates a feature vector extracted only from that landmark tracking result, as the expression evolves over time; whereas a red line connecting two landmarks indicates the feature vector extracted from that pair of landmarks tracking result, as the expression evolves over time.

Figure 6.

The first few (20, 40, and 60) features selected by AdaBoost for each class of facial expression. A red dot indicates a feature vector extracted from the tracking result of that landmark, as the facial expression evolves over time, whereas a red line connecting two landmarks indicates a feature vector extracted from the tracking result of those pair of landmarks, as the facial expression evolves over time.

Figure 7.

Grouping of landmarks into different regions according to the facial geometry.

Figure 8.

Recognition accuracy under different numbers of AdaBoost selected features.

Figure 9.

Average confusion score of week classifiers in percentages, for the recognition of each class of facial expressions.