Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2008 Nov 25:1242:162-71.
doi: 10.1016/j.brainres.2008.06.083. Epub 2008 Jun 28.

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

Julie N Buchan  1 , Martin Paré, Kevin G Munhall
Affiliations

The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception

Julie N Buchan et al. Brain Res. .

Abstract

During face-to-face conversation the face provides auditory and visual linguistic information, and also conveys information about the identity of the speaker. This study investigated behavioral strategies involved in gathering visual information while watching talking faces. The effects of varying talker identity and varying the intelligibility of speech (by adding acoustic noise) on gaze behavior were measured with an eyetracker. Varying the intelligibility of the speech by adding noise had a noticeable effect on the location and duration of fixations. When noise was present subjects adopted a vantage point that was more centralized on the face by reducing the frequency of the fixations on the eyes and mouth and lengthening the duration of their gaze fixations on the nose and mouth. Varying talker identity resulted in a more modest change in gaze behavior that was modulated by the intelligibility of the speech. Although subjects generally used similar strategies to extract visual information in both talker variability conditions, when noise was absent there were more fixations on the mouth when viewing a different talker every trial as opposed to the same talker every trial. These findings provide a useful baseline for studies examining gaze behavior during audiovisual speech perception and perception of dynamic faces.

PubMed Disclaimer

Figures

Figure 1
Figure 1. Histogram of all fixations by experimental condition
The distribution of fixation durations for each experimental condition is shown.
Figure 2
Figure 2. Overall number of fixations by region of interest
The overall number of fixations in each region of interest is shown. 2A shows gaze as a function of noise and 2B shows gaze as a function of talker variability. 2C shows the significant interaction between noise and talker variability on the number of gaze fixations made in the mouth region of interest. An asterisk denotes significant differences. The error bars indicate the standard errors of the mean.
Figure 3
Figure 3. Overall duration of fixations by region of interest
The median durations of fixations falling in each region of interest is shown. 3A shows gaze as a function of noise and 3B shows gaze as a function of talker variability. An asterisk denotes significant differences. The error bars indicate the standard errors of the mean.
Figure 4
Figure 4. Number of first fixations by region of interest
The number of first fixations in each trial falling in each region of interest is shown. 4A shows gaze as a function of noise and 4B shows gaze as a function of talker variability. An asterisk denotes significant differences. The error bars indicate the standard errors of the mean.
Figure 5
Figure 5. Number of second fixations by region of interest
The number of second fixations in each trial falling in each region of interest is shown. 5A shows gaze as a function of noise and 5B shows gaze as a function of talker variability. 5C shows the significant interaction between noise and talker variability on the number of gaze fixations made in the mouth region of interest. An asterisk denotes significant differences. The error bars indicate the standard errors of the mean.
Figure 6
Figure 6. Regions of interest
The ellipses indicate the regions of interest (the talker's right eye and left eye, nose and mouth) used in the analyses for four of the sixteen talkers are shown. Videos of the talkers were presented in color. See experimental procedures section for details.
See this image and copyright information in PMC

References

    1. Althoff RR, Cohen NJ. Eye-movement-based memory effect: a reprocessing effect in face perception. J Exp Psychol Learn. 1999;25:997–1010. - PubMed
    1. Ambadar Z, Schooler JW, Cohn JF. Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions. Psychol Sci. 2005;16:403–410. - PubMed
    1. Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I. The “Reading the mind in the eyes” test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. J Child Psychol Psychiatr. 2001;42:241–251. - PubMed
    1. Barton J, Radcliffe N, Cherkasova M, Edelman J, Intriligator J. Information processing during face recognition: the effects of familiarity, inversion and morphing on scanning fixations. Perception. 2006;353:1089–1105. - PubMed
    1. Benoît C, Guiard-Marigny T, Le Groff B, Adjoudani A. Which components of the face do humans and machines best speechread? In: Stork DG, Hennecke M, editors. Speechreading by Humans and Machines: Models, Systems & Applications. Kluwer Academic Publishers; 1996.

Publication types