Computerized measurement of facial expression of emotions in schizophrenia

Abstract

Deficits in the ability to express emotions characterize several neuropsychiatric disorders and are a hallmark of schizophrenia, and there is need for a method of quantifying expression, which is currently done by clinical ratings. This paper presents the development and validation of a computational framework for quantifying emotional expression differences between patients with schizophrenia and healthy controls. Each face is modeled as a combination of elastic regions, and expression changes are modeled as a deformation between a neutral face and an expressive face. Functions of these deformations, known as the regional volumetric difference (RVD) functions, form distinctive quantitative profiles of expressions. Employing pattern classification techniques, we have designed expression classifiers for the four universal emotions of happiness, sadness, anger and fear by training on RVD functions of expression changes. The classifiers were cross-validated and then applied to facial expression images of patients with schizophrenia and healthy controls. The classification score for each image reflects the extent to which the expressed emotion matches the intended emotion. Group-wise statistical analysis revealed this score to be significantly different between healthy controls and patients, especially in the case of anger. This score correlated with clinical severity of flat affect. These results encourage the use of such deformation based expression quantification measures for research in clinical applications that require the automated measurement of facial affect.

Figure 1

(a) Predefined template image of neutral expression. (b) Labeled template image. (c) Labeled subject facial expression image. These boundary labels identify the regions in which we consider change to occur during expression formation. Elastic registration is performed constrained by the corresponding regions.

Figure 2

Removal of inter-subject variability in the neutral face by computing deformations in the space of a common template. We use the information obtained from V_NE for all our analysis.

Figure 3

Expression displacement from neutral to happiness. The left two images are neutral (a) and happy (b) facial images. The expression displacement field (c) characterizes the motion of facial regions using vectors. The log RVD map (d) is a quantification of this displacement field and represents a spatial profile of the expansion and contraction of the facial regions, as part of the expression change.

Figure 4

Emotional expressions from the professional actors’ database. This database is used as the “normative standard” of emotional expression to train SVM based classifiers.

Figure 5

Facial images of healthy controls (top row) and schizophrenia patients (bottom row) expressing the four emotions of happiness, sadness, anger and fear.

Figure 6

For actors database, (a) cluster plot of C-SAFE score, vs. the percentage of the time the human raters correctly identified the emotion of expression, and (b) cluster plot of C-SAFE emotion score, vs. the rated intensity of the intended emotion. Each point in this plot corresponds to an expression in the actors’ database.

Figure 7

Examples from the database of patients and controls where the intended emotion did not subjectively match the facial expression but where the classifier determined the expression based on the facial expression well. The first was classified as uncertain, the second as sad, and the third as happy.

Figure 8

(a) Average C-SAFE Score vs. Video SANS Flatness, (b) Average C-SAFE Score vs. Video SANS Inappropriateness

Figure 9

Correlations by emotion and area of deficit. Anger is correlated with inappropriate affect and happiness is correlated with flat affect. All others are insignificant.