Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Feb 11:11:552533.
doi: 10.3389/fpsyg.2020.552533. eCollection 2020.

When Gesture "Takes Over": Speech-Embedded Nonverbal Depictions in Multimodal Interaction

Hui-Chieh Hsu  1 , Geert Brône  1 , Kurt Feyaerts  1
Affiliations

When Gesture "Takes Over": Speech-Embedded Nonverbal Depictions in Multimodal Interaction

Hui-Chieh Hsu et al. Front Psychol. .

Abstract

The framework of depicting put forward by Clark (2016) offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: "speech-embedded nonverbal depictions," namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples from American TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance.

Keywords: depiction; embedding; gesture; iconicity; multimodality.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

FIGURE 1
FIGURE 1
Depiction in (1).
FIGURE 2
FIGURE 2
Depictions in (6).
FIGURE 3
FIGURE 3
Depictions in (7).
FIGURE 4
FIGURE 4
Depiction in (8).
FIGURE 5
FIGURE 5
Continuum of information contribution from non-depictive speech and depictive signals.
FIGURE 6
FIGURE 6
Depictions in (9).
FIGURE 7
FIGURE 7
Depiction in (10).
FIGURE 8
FIGURE 8
Depiction in (12).
FIGURE 9
FIGURE 9
Depiction in (13).
FIGURE 10
FIGURE 10
Depiction in (18).
FIGURE 11
FIGURE 11
Depiction in (19).
FIGURE 12
FIGURE 12
Depiction in (20).
FIGURE 13
FIGURE 13
Depiction in (21).
FIGURE 14
FIGURE 14
Depiction in (22).
FIGURE 15
FIGURE 15
Depiction in (23).
FIGURE 16
FIGURE 16
Depictions in (25).
FIGURE 17
FIGURE 17
Depictions in (27).
APPENDIX
APPENDIX
Screenshot of annotation in ELAN (of Example 27).
See this image and copyright information in PMC

References

    1. Arbib M. A. (2005). From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics. Behav. Brain Sci. 28 105–124. 10.1017/S0140525X05000038 - DOI - PubMed
    1. Arnheim R. (1969). Visual Thinking. Berkeley: University of California Press.
    1. Bavelas J. B., Chovil N. (2000). Visible acts of meaning: an integrated message model of language in face-to-face dialogue. J. Lang. Soc. Psychol. 19 163–194. 10.1177/0261927X00019002001 - DOI
    1. Bavelas J. B., Gerwing J., Healing S. (2014). Effect of dialogue on demonstrations: direct quotations, facial portrayals, hand gestures, and figurative references. Discourse Process. 51 619–655. 10.1080/0163853X.2014.883730 - DOI
    1. Beukeleers I., Hsu H.-C. (2019). “Complex composite depictions and their semiotic diversity: evidence from gestures and signs,” Paper Presented at the 6th European and 9th Nordic Symposium on Multimodal Communication, Leuven.

LinkOut - more resources