Modality-Specific Perceptual Learning of Vocoded Auditory versus Lipread Speech: Different Effects of Prior Information

Abstract

Traditionally, speech perception training paradigms have not adequately taken into account the possibility that there may be modality-specific requirements for perceptual learning with auditory-only (AO) versus visual-only (VO) speech stimuli. The study reported here investigated the hypothesis that there are modality-specific differences in how prior information is used by normal-hearing participants during vocoded versus VO speech training. Two different experiments, one with vocoded AO speech (Experiment 1) and one with VO, lipread, speech (Experiment 2), investigated the effects of giving different types of prior information to trainees on each trial during training. The training was for four ~20 min sessions, during which participants learned to label novel visual images using novel spoken words. Participants were assigned to different types of prior information during training: Word Group trainees saw a printed version of each training word (e.g., "tethon"), and Consonant Group trainees saw only its consonants (e.g., "t_th_n"). Additional groups received no prior information (i.e., Experiment 1, AO Group; Experiment 2, VO Group) or a spoken version of the stimulus in a different modality from the training stimuli (Experiment 1, Lipread Group; Experiment 2, Vocoder Group). That is, in each experiment, there was a group that received prior information in the modality of the training stimuli from the other experiment. In both experiments, the Word Groups had difficulty retaining the novel words they attempted to learn during training. However, when the training stimuli were vocoded, the Word Group improved their phoneme identification. When the training stimuli were visual speech, the Consonant Group improved their phoneme identification and their open-set sentence lipreading. The results are considered in light of theoretical accounts of perceptual learning in relationship to perceptual modality.

Keywords: lipreading; multisensory; perceptual learning; speech perception; speech perception training; spoken language processing; vocoded speech; word learning.

Figure 1

Outline of the Novel Word Training and Testing. When prior information was presented, it was presented first on each trial, followed by the speech stimulus. Then, a matrix of 12 nonsense pictures was displayed, and the participant clicked on a picture to label the word. If the selection was correct, it was highlighted; if it was incorrect, the correct picture was highlighted. The spoken stimulus was then repeated, and the participant selected the correct picture to move on to the next trial. The test that followed each training set comprised only the stimulus followed by the matrix of pictures and the participant’s response. There was no prior information, and the trial terminated with the participant’s response, without any feedback. Additionally, half of the training words were replaced by foils.

Figure 2

Examples of natural and vocoded speech. Beneath each natural stimulus (A,C) is its vocoded version (B,D, respectively).

Figure 3

Outline of Experiment 1 procedures. Pre- and posttraining sessions and training sessions were carried out on different days.

Figure 4

Effects plot: Modeled vocoder training. The fixed effects are shown for the individual groups, sessions, and blocks. The scale is proportion of correct words.

Figure 5

Fixed-Effects Plots: Modeled Retention Results for Vocoded Novel Word Labeling in Experiment 1.

Figure 6

Fixed-Effects Plot: Modeled pre- and posttraining forced-choice consonant identification. The effects are shown for the individual training groups and the Control Group across Pre- and Posttraining sessions and for the different consonant positions.

Figure 7

Outline of Experiment 2 procedures. Pre- and posttraining and novel word training were carried out in different sessions. Phoneme identification and open-set sentence recognition were tested in the same sessions.

Figure 8

Fixed-Effects plot: Modeled Novel Word Lipreading Training. The effects for lipreading training trials in Block 3 are shown for each of the training groups.

Figure 9

Fixed-Effects Plots: Modeled Novel Word Retention Results for Lipread Novel Word Labeling in Experiment 2.

Figure 10

Effects Plot: Modeled Visual Speech Forced-Choice Consonant Identification. The effects are shown for the individual training groups and the Control Group across Pre- and Posttraining.

Figure 11

Fixed-Effects Plot: Modeled Sentence Lipreading. The effects are shown for the individual training groups and the Control Group across Pre- and Posttraining sessions.

Figure 12

Fixed-Effects Plot: Comparison of Novel Word Retention by Word versus Consonant Groups Across Experiments 1 and 2. Session was coded as a continuous variable to obtain the smooth lines.