Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Nov 29;16(4):10.16910/jemr.16.4.4.
doi: 10.16910/jemr.16.4.4. eCollection 2023.

Primacy of mouth over eyes to perceive audiovisual Mandarin lexical tones

Biao Zeng  1 , Guoxing Yu  2 , Nabil Hasshim  3 , Shanhu Hong  4
Affiliations

Primacy of mouth over eyes to perceive audiovisual Mandarin lexical tones

Biao Zeng et al. J Eye Mov Res. .

Abstract

The visual cues of lexical tones are more implicit and much less investigated than consonants and vowels, and it is still unclear what facial areas contribute to facial tones identification. This study investigated Chinese and English speakers' eye movements when they were asked to identify audiovisual Mandarin lexical tones. The Chinese and English speakers were presented with an audiovisual clip of Mandarin monosyllables (for instance, /ă/, /à/, /ĭ/, /ì/) and were asked to identify whether the syllables were a dipping tone (/ă/, / ĭ/) or a falling tone (/ à/, /ì/). These audiovisual syllables were presented in clear, noisy and silent (absence of audio signal) conditions. An eye-tracker recorded the participants' eye movements. Results showed that the participants gazed more at the mouth than the eyes. In addition, when acoustic conditions became adverse, both the Chinese and English speakers increased their gaze duration at the mouth rather than at the eyes. The findings suggested that the mouth is the primary area that listeners utilise in their perception of audiovisual lexical tones. The similar eye movements between the Chinese and English speakers imply that the mouth acts as a perceptual cue that provides articulatory information, as opposed to social and pragmatic information.

Keywords: Chinese speakers; English speakers; audiovisual speech; eye movement; gaze; lexical tone.

PubMed Disclaimer

Conflict of interest statement

The author(s) declare(s) that the contents of the article are in agreement with the ethics described in http://biblio.unibe.ch/portale/elibrary/BOP/jemr/ethics.html and that there is no conflict of interest regarding the publication of this paper.

Figures

Figure 1.
Figure 1.
Face in the left side of a screen and takes 2/3 of this full screen. Red colour indicates longer gaze duration and the green colour shorter gaze duration.
Figure 2.
Figure 2.
Mean number of fixations on each ROI.
Figure 3.
Figure 3.
Percentage of gaze duration on each ROI.
See this image and copyright information in PMC

References

    1. Alsius, A., Paré, M., & Munhall, K. G. (2018). Forty years after hearing lips and seeing voices: The McGurk effect revisited. Multisensory Research, 31(1-2), 111–144. 10.1163/22134808-00002565 - DOI - PubMed
    1. Best, V., Ozmeral, E. J., & Shinn-Cunningham, B. G. (2007). Visually-guided attention enhances target identification in a complex auditory scene. Journal of the Association for Research in Otolaryngology, 8(2), 294–304. 10.1007/s10162-007-0073-z - DOI - PMC - PubMed
    1. Buchan, J. N., Paré, M., & Munhall, K. G. (2008). The effect of varying talker identity and listening conditions on gaze behavior during audiovisual speech perception. Brain Research, 1242, 162–171. 10.1016/j.brainres.2008.06.083 - DOI - PMC - PubMed
    1. Burnham, D., Reynolds, J., Vatikiotis-Bateson, E., Yehia, H., Ciocca, V., Morris, R. H., et al. Jones, C. (2006). The perception and production of phones and tones: The role of rigid and non-rigid face and head motion. https://ro.uow.edu.au/cgi/viewcontent.cgi?article=1676&context=edupapers
    1. Burnham, D., Kasisopa, B., Reid, A., Luksaneeyanawin, S., Lacerda, F., Attina, V., Rattanasone, N. X., Schwarz, I.-C., & Webster, D. (2015). Universality and language-specific experience in the perception of lexical tone and pitch. Applied Psycholinguistics, 36(6), 1459–1491. 10.1017/S0142716414000496 - DOI

LinkOut - more resources