Neural decoding of emotional prosody in voice-sensitive auditory cortex predicts social communication abilities in children

Abstract

During social interactions, speakers signal information about their emotional state through their voice, which is known as emotional prosody. Little is known regarding the precise brain systems underlying emotional prosody decoding in children and whether accurate neural decoding of these vocal cues is linked to social skills. Here, we address critical gaps in the developmental literature by investigating neural representations of prosody and their links to behavior in children. Multivariate pattern analysis revealed that representations in the bilateral middle and posterior superior temporal sulcus (STS) divisions of voice-sensitive auditory cortex decode emotional prosody information in children. Crucially, emotional prosody decoding in middle STS was correlated with standardized measures of social communication abilities; more accurate decoding of prosody stimuli in the STS was predictive of greater social communication abilities in children. Moreover, social communication abilities were specifically related to decoding sadness, highlighting the importance of tuning in to negative emotional vocal cues for strengthening social responsiveness and functioning. Findings bridge an important theoretical gap by showing that the ability of the voice-sensitive cortex to detect emotional cues in speech is predictive of a child's social skills, including the ability to relate and interact with others.

Keywords: development; emotion recognition; speech; superior temporal sulcus; voice.

Fig. 1

fMRI stimuli, scanning protocol, and analysis strategy. A) fMRI stimuli consisted of acoustic sentences spoken in emotional and neutral prosody. Spectrograms of sentence #2, “my spoon is on the table,” spoken in neutral speech (upper panel), in sad prosody (lower-left panel) and in happy prosody (lower-right panel). B) Stimuli were selected based on results from a behavioral experiment conducted in an independent cohort of 27 school-age, typically developing children and 9 adults, who provided ratings on a 5-point scale (“how sad or happy is this voice?”). C) A sparse sampling fMRI scanning protocol with a repetition time (TR) larger than the acquisition time (TA) was used to present acoustic stimuli during silent intervals between volume acquisitions to eliminate the effects of scanner noise on auditory perception. D) Schematic of the analyses employed in the study. (i) Definition of ROIs within auditory cortex. (ii) ROI-based multivariate pattern classification of neutral and emotional prosody was employed to compare decoding accuracy between auditory cortical subdivisions of the STP and STS. (iii) A whole-brain multivariate pattern classification method was used to examine whether brain regions beyond auditory cortex accurately discriminate emotional and neutral prosody stimuli. (iv) Associations between children’s neural decoding of emotional prosody, social communication skills, and emotion recognition accuracy.

Fig. 2

ROI-based emotional prosody decoding. A) Auditory cortical brain regions included in the ROI-based emotional prosody decoding analysis. ROIs located on the STP (HG, PT, and PP) are colored in green, while ROIs located within STS (pSTS, mSTS, and aSTS) are colored in blue. B) Classification accuracies for bilateral STP and STS ROIs for the (sad prosody versus neutral speech) and (happy prosody versus neutral speech) contrasts. Results show consistently greater emotional prosody decoding in the STS compared to the STP across contrasts and hemispheres. C) Classification accuracies for all bilateral auditory cortical regions. Post hoc comparisons showed that classification accuracy within mSTS is consistently greater than those measured in HG, which served as a reference region for this analysis. Classification accuracy within pSTS was also greater than HG for 3 of the stimulus contrasts. Abbreviations: ^* = P < 0.05; ^** = P < 0.01; ^*** = P < 0.001; HG = Heschl’s gyrus; LH = left hemisphere; PP = planum polare; PT = planum temporale; RH = right hemisphere.

Fig. 3

Searchlight-based prosody decoding. Chance level for the 2-class emotional prosody decoding was at 50%. A) For the (sad prosody versus neutral speech) contrast, the whole-brain searchlight identified a significant cluster in the right pSTS. No significant clusters outside of superior temporal cortex were identified for the (sad prosody versus neutral speech) contrast. B) For the (happy prosody versus neutral speech) contrast, the whole-brain analysis identified multiple clusters in bilateral STS and STG. In addition, a large cluster in the left central operculum was identified. Abbreviations: FWE = family-wise error.

Fig. 4

Neural decoding of sad prosody predicts social communication abilities. Results revealed a significant positive association between decoding of the (sad prosody versus neutral speech) contrast in the mSTS and social communication abilities. Specifically, more accurate emotional prosody decoding in the mSTS was associated with greater social communication abilities in children.