The Multisensory Nature of Verbal Discourse in Parent-Toddler Interactions

Abstract

Toddlers learn object names in sensory rich contexts. Many argue that this multisensory experience facilitates learning. Here, we examine how toddlers' multisensory experience is linked to another aspect of their experience associated with better learning: the temporally extended nature of verbal discourse. We observed parent-toddler dyads as they played with, and as parents talked about, a set of objects. Analyses revealed links between the multisensory and extended nature of speech, highlighting inter-connections and redundancies in the environment. We discuss the implications of these results for our understanding of early discourse, multisensory communication, and how the learning environment shapes language development.

Figure 1.

Example image frames from toddler egocentric cameras found to be associated with object name learning (see Pereira et al., 2014).

Figure 2.

The experimental set-up: Toddlers equipped with head cameras and their parents played with mono-colored objects in a laboratory room; play sessions were recorded from multiple angles.

Figure 3.

Illustration of the automated processing of a single head camera image frame (A–C). Sampling of objects that were classified as visually dominant using the 5% field of view threshold (D).

Figure 4.

Representative time series of parents’ referential utterances (top row) and the nonverbal events (bottom three rows); colors reflect the target object of speech and nonverbal event.

Figure 5.

Example time series of parents’ utterances to different objects over the course of a trial. Adjacent utterances to the same object were merged into episodes of discourse.

Figure 6.

Mean proportion of utterances within extended discourse episodes that overlapped with each nonverbal property (A–C), and that were classified as at least bimodal (D) or multimodal (E).

Figure 7.

Hypothetical data (top row) and predictions (bottom row) for two hypotheses on the synchronous nature of the multisensory coupling in parents’ speech. Highly synchronous coupling would lead to greater overlap with nonverbal events in speech versus nonspeech segments; less synchronous coupling would lead to greater parity in overlap with nonverbal events between speech and nonspeech segments.

Figure 8.

Illustration of the relevant windows of analysis for speech and nonspeech segments (A). Mean proportion of speech and nonspeech segments that overlapped with the three nonverbal properties (B–D).

Figure 9.

Example time series of discourse episodes depicting the coupling between verbal and each nonverbal property.

Figure 10.

(A) Examples of relevant moments for analysis for utterances in extended and brief episodes; (B–D) Mean proportion of utterances that contained some overlap with the three nonverbal properties.