Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Mar 3;31(1):57-77.
doi: 10.1044/2021_AJA-21-00034. Epub 2021 Dec 29.

During Lipreading Training With Sentence Stimuli, Feedback Controls Learning and Generalization to Audiovisual Speech in Noise

Lynne E Bernstein  1 , Edward T Auer  1 , Silvio P Eberhardt  1
Affiliations

During Lipreading Training With Sentence Stimuli, Feedback Controls Learning and Generalization to Audiovisual Speech in Noise

Lynne E Bernstein et al. Am J Audiol. .

Abstract

Purpose: This study investigated the effects of external feedback on perceptual learning of visual speech during lipreading training with sentence stimuli. The goal was to improve visual-only (VO) speech recognition and increase accuracy of audiovisual (AV) speech recognition in noise. The rationale was that spoken word recognition depends on the accuracy of sublexical (phonemic/phonetic) speech perception; effective feedback during training must support sublexical perceptual learning.

Method: Normal-hearing (NH) adults were assigned to one of three types of feedback: Sentence feedback was the entire sentence printed after responding to the stimulus. Word feedback was the correct response words and perceptually near but incorrect response words. Consonant feedback was correct response words and consonants in incorrect but perceptually near response words. Six training sessions were given. Pre- and posttraining testing included an untrained control group. Test stimuli were disyllable nonsense words for forced-choice consonant identification, and isolated words and sentences for open-set identification. Words and sentences were VO, AV, and audio-only (AO) with the audio in speech-shaped noise.

Results: Lipreading accuracy increased during training. Pre- and posttraining tests of consonant identification showed no improvement beyond test-retest increases obtained by untrained controls. Isolated word recognition with a talker not seen during training showed that the control group improved more than the sentence group. Tests of untrained sentences showed that the consonant group significantly improved in all of the stimulus conditions (VO, AO, and AV). Its mean words correct scores increased by 9.2 percentage points for VO, 3.4 percentage points for AO, and 9.8 percentage points for AV stimuli.

Conclusions: Consonant feedback during training with sentences stimuli significantly increased perceptual learning. The training generalized to untrained VO, AO, and AV sentence stimuli. Lipreading training has potential to significantly improve adults' face-to-face communication in noisy settings in which the talker can be seen.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
Outline of the three feedback types. Figure 1a gives an example of sentence feedback. Figure 1b gives two examples of word feedback. Figure 1c gives two examples of consonant feedback.
Figure 2.
Figure 2.
Time series for training by each participant in each feedback group. (a) The top panel is scored as mean proportion words correct, and (b) the bottom panel is scores as mean proportion phonemes correct. Plotted with ggplot2 in R (Wickham, 2016).
Figure 3.
Figure 3.
Mean proportion consonants correct per position (initial, medial, and final) in CVCVC (C = consonant; V = vowel) stimuli during pre- and posttraining forced-choice identification. Error bars are standard error of the means. Plotted with ggplot2 in R (Wickham, 2016).
Figure 4.
Figure 4.
Effects plot for means in mixed-model analyses of responses to isolated sentences presented in VO, AV, and AO conditions during pre- and posttraining test sessions. The plot shows the interaction of test session with condition across groups. (a) Results are plotted for proportion words correct scoring. (b) Results are plotted for proportion phonemes correct scoring. Plotted with the effects library in R (Fox & Weisberg, 2019). VO = visual-only; AO = auditory-only; AV = audiovisual.
Figure 5.
Figure 5.
Modeled two-way interactions between group and pre- and posttraining test sessions with sentence stimuli presented in VO, AV, and AO conditions. The results were scored as proportion phonemes correct. The positions noted with a significance level indicate a significant Group × Test Session interaction relative to the consonant group. *p < .05; **p < .01; ***p < .001. Plotted with the effects library in R (Fox & Weisberg, 2019). VO = visual-only; AO = auditory-only; AV = audiovisual.
Figure 6.
Figure 6.
Scatter plots, histograms, and correlations for pre- and posttraining sentence responses. The figure used the mean proportion words correct scores for the isolated sentences in the stimulus conditions VO, AO, and AV, per participant for all of the participants tested at pre and posttraining test sessions. Pearson correlations are shown. Pre_VO, Pre_AV, and Pre, AO = proportion words correct scores at pretesting. Post_VO, Post _AV, and Post_AO = proportion words correct scores at posttesting. ***p < .001; *p < .05. Plotted with Performance Analytics in R (2019). VO = visual-only; AO = auditory-only; AV = audiovisual.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Ahissar, M. , Nahum, M. , Nelken, I. , & Hochstein, S. (2009). Reverse hierarchies and sensory learning. Philosophical Transactions of the Royal Society B, 364(1515), 285–299. https://doi.org/10.1098/rstb.2008.0253 - PMC - PubMed
    1. Ashby, F. G. , & Maddox, W. T. (2011). Human category learning 2.0. Annals of the New York Academy of Sciences, 1224(1), 147–161. https://doi.org/10.1111/j.1749-6632.2010.05874.x - PMC - PubMed
    1. Ashby, F. G. , & Valentin, V. V. (2017). Chapter 7 - Multiple systems of perceptual category learning: Theory and cognitive tests. In Cohen H. & Lefebvre C. (Eds.), Handbook of categorization in cognitive science (2nd ed., pp. 157–188). Elsevier. https://doi.org/10.1016/B978-0-08-101107-2.00007-5
    1. Ashby, F. G. , & Vucovich, L. E. (2016). The role of feedback contingency in perceptual category learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(11), 1731–1746. https://doi.org/10.1037/xlm0000277 - PMC - PubMed
    1. Auer, E. T., Jr. , & Bernstein, L. E. (1997). Speechreading and the structure of the lexicon: Computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness. The Journal of the Acoustical Society of America, 102(6), 3704–3710. https://doi.org/10.1121/1.420402 - PubMed

LinkOut - more resources