The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy

Abstract

In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.

Keywords: Audiovisual integration; Infant; Neurocognitive development; Speech processing; fNIRS.

Fig. 1

FNIRS headgear and experimental paradigm. (a) Picture of an infant wearing the NTS fNIRS headgear and illustrations of channel location in relation to infant’s head: highlighted sources (stars) and detectors (diamonds), grey circles indicate measurement channels and channel numbers with the 10–20 coordinates superimposed on the diagram in green. Channels within the yellow box are part of the inferior frontal region, orange - superior temporal region. (b) Experimental stimuli design and study paradigm

Fig. 2

Theoretical pattern of predicted non-linear responses to the alternating unimodal (striped) and bimodal (plain) conditions. If the difference between the conditions was significant, responses were interpreted as either super- or sub-additive. Note that interpretation of the response differed depending on the chromophore: HbO or HbR

Fig. 3

Mean concentration of chromophores (µM, red - HbO, blue - HbR) in the second time window (10-15s) depending on condition: striped - alternating unimodal (auditory + visual) speech, plain - bimodal (audiovisual) speech. Top panel - younger age group, bottom panel - older age group. * p <. 05