Finding Phrases: The Interplay of Word Frequency, Phrasal Prosody and Co-speech Visual Information in Chunking Speech by Monolingual and Bilingual Adults

Abstract

The audiovisual speech signal contains multimodal information to phrase boundaries. In three artificial language learning studies with 12 groups of adult participants we investigated whether English monolinguals and bilingual speakers of English and a language with opposite basic word order (i.e., in which objects precede verbs) can use word frequency, phrasal prosody and co-speech (facial) visual information, namely head nods, to parse unknown languages into phrase-like units. We showed that monolinguals and bilinguals used the auditory and visual sources of information to chunk "phrases" from the input. These results suggest that speech segmentation is a bimodal process, though the influence of co-speech facial gestures is rather limited and linked to the presence of auditory prosody. Importantly, a pragmatic factor, namely the language of the context, seems to determine the bilinguals' segmentation, overriding the auditory and visual cues and revealing a factor that begs further exploration.

Keywords: artificial grammar learning; bilingualism; co-speech visual information; frequency-based information; phrase segmentation; prosody.

Figure 1.

Shared structure of the artificial languages. The table represents the shared basic structure of the ambiguous artificial languages: (a) the lexical categories and tokens of the languages; (b) the two possible structures of the ambiguous stream; (c) three examples of the 36 test pairs. On the right, a picture of the animated line drawing used in the languages containing visual information.

Figure 2.

Graphical depiction of the artificial languages in Experiments 1, 2 and 3. The brackets depict the duration of the head nods, whereas the arrows signal the location of their peak.

Figure 3.

Word order preferences of the participants in Experiment 1. Bar graphs (top) and boxplots (bottom) with standard error depicting the number and distribution of frequent-initial responses out of the 36 test trials by the monolingual (dark gray columns) and bilingual (light gray columns) participants.

Figure 4.

Word order preferences of the participants in Experiment 2. Bar graphs (top) and boxplots (bottom) with standard error depicting the number and distribution of frequent-initial responses out of the 36 test trials by the monolingual (dark gray columns) and bilingual (light gray columns) participants. Note that the patterned columns in the top figure depict Exp.1’s groups exposed to frequency-only information, that is, the baseline groups. Experiment 1 and 2’s artificial languages share the same tokens and test items.

Figure 5.

Word order preferences of the participants in Experiment 3. Bar graphs (top) and boxplots (bottom) with standard error depicting the number and distribution of frequent-initial responses out of the 36 test trials by the monolingual (dark gray columns) and bilingual (light gray columns) participants. The patterned columns in the top figure depict Exp.1’s groups exposed to frequency-only information, that is, the baseline groups. Experiment 1, 2 and 3’s artificial languages share the same tokens and test items.