Use of audiovisual information in speech perception by prelingually deaf children with cochlear implants: a first report

Abstract

Objective: Although there has been a great deal of recent empirical work and new theoretical interest in audiovisual speech perception in both normal-hearing and hearing-impaired adults, relatively little is known about the development of these abilities and skills in deaf children with cochlear implants. This study examined how prelingually deafened children combine visual information available in the talker's face with auditory speech cues provided by their cochlear implants to enhance spoken language comprehension.

Design: Twenty-seven hearing-impaired children who use cochlear implants identified spoken sentences presented under auditory-alone and audiovisual conditions. Five additional measures of spoken word recognition performance were used to assess auditory-alone speech perception skills. A measure of speech intelligibility was also obtained to assess the speech production abilities of these children.

Results: A measure of audiovisual gain, "Ra," was computed using sentence recognition scores in auditory-alone and audiovisual conditions. Another measure of audiovisual gain, "Rv," was computed using scores in visual-alone and audiovisual conditions. The results indicated that children who were better at recognizing isolated spoken words through listening alone were also better at combining the complementary sensory information about speech articulation available under audiovisual stimulation. In addition, we found that children who received more benefit from audiovisual presentation also produced more intelligible speech, suggesting a close link between speech perception and production and a common underlying linguistic basis for audiovisual enhancement effects. Finally, an examination of the distribution of children enrolled in Oral Communication (OC) and Total Communication (TC) indicated that OC children tended to score higher on measures of audiovisual gain, spoken word recognition, and speech intelligibility.

Conclusions: The relationships observed between auditory-alone speech perception, audiovisual benefit, and speech intelligibility indicate that these abilities are not based on independent language skills, but instead reflect a common source of linguistic knowledge, used in both perception and production, that is based on the dynamic, articulatory motions of the vocal tract. The effects of communication mode demonstrate the important contribution of early sensory experience to perceptual development, specifically, language acquisition and the use of phonological processing skills. Intervention and treatment programs that aim to increase receptive and productive spoken language skills, therefore, may wish to emphasize the inherent cross-correlations that exist between auditory and visual sources of information in speech perception.

Figure 1

Top panel shows the frequency distribution of R_a scores (enhancement relative to auditory-alone performance) for the sample. Bottom panel shows the frequency distribution of R_v scores (enhancement relative to visual-alone performance). Higher R scores denote larger gains in accuracy in the audiovisual condition relative to accuracy in the relevant unimodal condition.

Figure 2

Audiovisual benefit (R_a) for the high-performing (above the median) and low-performing (below the median) groups of the median splits for five measures of auditory-alone word recognition. Error bars are standard errors.

Figure 3

Scatterplots showing the relationships between R_a score (audiovisual benefit) and the auditory-alone measures of spoken word recognition

Figure 4

Scatterplots showing the relationship between visual enhancement and speech intelligibility split by communication mode. Oral Communication children are represented by open triangles; Total Communication children are represented by filled triangles.