Facial Expressions as an Index of Listening Difficulty and Emotional Response

Abstract

Knowledge about listening difficulty experienced during a task can be used to better understand speech perception processes, to guide amplification outcomes, and can be used by individuals to decide whether to participate in communication. Another factor affecting these decisions is individuals' emotional response which has not been measured objectively previously. In this study, we describe a novel method of measuring listening difficulty and affect of individuals in adverse listening situations using automatic facial expression algorithm. The purpose of our study was to determine if facial expressions of confusion and frustration are sensitive to changes in listening difficulty. We recorded speech recognition scores, facial expressions, subjective listening effort scores, and subjective emotional responses in 33 young participants with normal hearing. We used the signal-to-noise ratios of -1, +2, and +5 dB SNR and quiet conditions to vary the difficulty level. We found that facial expression of confusion and frustration increased with increase in difficulty level, but not with change in each level. We also found a relationship between facial expressions and both subjective emotion ratings and subjective listening effort. Emotional responses in the form of facial expressions show promise as a measure of affect and listening difficulty. Further research is needed to determine the specific contribution of affect to communication in challenging listening environments.

Keywords: confusion; emotional responses; facial expressions; listening effort.

Figure 1

Time course of the first trial of a speech in noise condition. Here the noise starts at 0 seconds. The first second of noise (or quiet in quite condition) is considered the baseline for the facial expression analyses. This is followed by 4 seconds of sentences that are offset-aligned. Then participants wait for 2 seconds (retention period). At the end of these 2 seconds, a prompt is given after which participants repeat the sentence.

Figure 2

Average speech recognition scores from the most (−1 dB signal to noise ratio [SNR]) to least difficulty (Quiet) level. The main effect of SNR is significant and all pairwise comparisons were significant. Levels of significance: *< 0.05; ** < 0.01; *** < 0.001.

Figure 3

Time course of the evidence of confusion (averaged across sentences and participants) across the different signal to noise ratios and in quiet. Sentence and noise are present from 0 to 4 seconds. 4 to 6 seconds is the retention period.

Figure 4

Area under the curve (AUC) for confusion (averaged across sentences and participants) for quiet and various signal to noise ratios (SNRs). The main effect of SNR is significant and the pairwise comparisons indicated in the figure were significant. Levels of significance: *< 0.05; ** < 0.01; *** < 0.001.

Figure 5

Correlation between the evidence of confusion and frustration (objective measures of emotional response represented using their area under the curve or AU). We find that they are positively related. Shapes correspond to the following signal to noise ratio (SNR): −1 dB (diamond), +2 dB (Square), +5 dB (circle), and quiet (triangle).

Figure 6

Listening effort rating for the different signal to noise ratios (SNRs) and quiet condition. The rating scale goes from 0 representing greatest listening effort to 10 representing least listening effort. These scores are an average across blocks of five sentences with four ratings in each condition from each participant. The main effect of SNR is significant and so are all the pairwise comparisons. Levels of significance: *< 0.05, ** < 0.01, *** < 0.001.

Figure 7

Subjective confusion rating for quiet and the different signal to noise ratios. A rating of 1 represents not at all confused to 5 representing very strong confusion. These scores are an average across blocks of five sentences with four ratings in each condition from each participant. The main effect of condition and all pairwise comparison are significant. Levels of significance: * < 0.05, ** < 0.01, *** < 0.001.

Figure 8

Relationship of subjective self-rating of confusion and objective area under the curve for confusion. Shapes correspond to the following signal to noise ratio (SNR): −1 dB (diamond), +2 dB (square), +5 dB (circle), and quiet (triangle).

Figure 9

Relationship of listening effort rating and objective area under the curve for confusion. Shapes correspond to the following signal to noise ratio (SNR): −1 dB (diamond), +2 dB (square), +5 dB (circle), and quiet (triangle).