The effect of lip-reading on primary stream segregation

Abstract

Lip-reading has been shown to improve the intelligibility of speech in multitalker situations, where auditory stream segregation naturally takes place. This study investigated whether the benefit of lip-reading is a result of a primary audiovisual interaction that enhances the obligatory streaming mechanism. Two behavioral experiments were conducted involving sequences of French vowels that alternated in fundamental frequency. In Experiment 1, subjects attempted to identify the order of items in a sequence. In Experiment 2, subjects attempted to detect a disruption to temporal isochrony across alternate items. Both tasks are disrupted by streaming, thus providing a measure of primary or obligatory streaming. Visual lip gestures articulating alternate vowels were synchronized with the auditory sequence. Overall, the results were consistent with the hypothesis that visual lip gestures enhance segregation by affecting primary auditory streaming. Moreover, increases in the naturalness of visual lip gestures and auditory vowels, and corresponding increases in audiovisual congruence may potentially lead to increases in the effect of visual lip gestures on streaming.

Figure 1

Schematic representation of an audiovisual sequence. Lip gestures were presented that articulated either the three high-pitch vowels or the three low- pitch vowels, selected randomly across trials [except that in some cases, F0(1) = F0(2)].

Figure 2

Synchronization of audio and visual streams. The figure shows lip movements congruent with the F0(2) vowels. The maximum opening of the mouth is centered on the cued audio vowel, but begins slightly before and remains somewhat after each cued auditory vowel. Each picture of the lips was extracted from the lip gesture movie that started 67 ms (i.e., two frames) before the corresponding auditory vowel and ended 100 ms after the offset of the auditory vowel (i.e., three frames).

Figure 3

Results from Experiment 1. Percent correct naming on the order of presentation of the six vowels is shown as a function of the F0 difference between alternate vowels and the visual condition. Bars represent the standard errors.

Figure 4

Schematic representation of an audiovisual sequence. The sequence is first regular and then the audio-only stream (in black) is gradually delayed to reach the final value of dT that the listener has to detect. The audiovisual stream (in gray) is always regular. Vertical dotted lines exhibit the transition from a regular to an irregular rhythm of the audio-only stream.

Figure 5

Results of Experiment 2. The threshold for detecting a temporal offset between streams is plotted for each F0 difference and each visual condition across the eight participants. Errors bars represent one standard error.

Figure 6

Visual lip gestures in Experiment 1.

Figure 7

Visual lip gestures in Experiment 2.