Exploring Hierarchical Auditory Representation via a Neural Encoding Model

Abstract

By integrating hierarchical feature modeling of auditory information using deep neural networks (DNNs), recent functional magnetic resonance imaging (fMRI) encoding studies have revealed the hierarchical neural auditory representation in the superior temporal gyrus (STG). Most of these studies adopted supervised DNNs (e.g., for audio classification) to derive the hierarchical feature representation of external auditory stimuli. One possible limitation is that the extracted features could be biased toward discriminative features while ignoring general attributes shared by auditory information in multiple categories. Consequently, the hierarchy of neural acoustic processing revealed by the encoding model might be biased toward classification. In this study, we explored the hierarchical neural auditory representation via an fMRI encoding framework in which an unsupervised deep convolutional auto-encoder (DCAE) model was adopted to derive the hierarchical feature representations of the stimuli (naturalistic auditory excerpts in different categories) in fMRI acquisition. The experimental results showed that the neural representation of hierarchical auditory features is not limited to previously reported STG, but also involves the bilateral insula, ventral visual cortex, and thalamus. The current study may provide complementary evidence to understand the hierarchical auditory processing in the human brain.

Keywords: deep convolutional auto-encoder; fMRI; hierarchical auditory representation; naturalistic experience; neural encoding.

FIGURE 1

The schematic illustration of the study. (A) fMRI acquisition using naturalistic auditory excerpts as stimuli. (B) Hierarchical feature representation of the naturalistic auditory stimuli via an unsupervised DCAE model. (C) Hierarchical acoustic features were correlated to fMRI brain activities using an encoding model based on LASSO to infer hierarchical auditory representation in the brain.

FIGURE 2

The DCAE model used for the hierarchical feature modeling.

FIGURE 3

The architecture of the supervised DNNs for audio classification. GAP, global average pooling.

FIGURE 4

Visualization of learned filters in the DCAE model. (A) Examples of the learned filters in each layer. (B) The power-spectrum patterns of learned filters. The x-axis represents the index of filters, the y-axis represents the frequency ranging from 0 to 8000 Hz.

FIGURE 5

Encoding performance of the trained DCAE model. (A) The distribution of Pearson correlation coefficients (PCC) between the input audio signals and reconstructed ones in the MagnaTagATune dataset and LibriSpeech Corpus. (B) The distribution of PCC in the auditory samples from the fMRI stimuli.

FIGURE 6

The encoding performance for each layer in the unsupervised DCAE model (A) and supervised classification model (B).

FIGURE 7

The comparison of encoding performance between the unsupervised DCAE model and supervised classification model in each layer. A1, primary auditory cortex; aSTG, anterior superior temporal gyrus; pSTG, posterior superior temporal gyrus; MTG, middle temporal gyrus; VC, visual cortex; PFC, prefrontal cortex.

FIGURE 8

Brain regions that are selectively activated by the hierarchical acoustic features represented in each encoder layer of the unsupervised DCAE model. Panels (A–D) represent the first four layers in the unsupervised DCAE model.

FIGURE 9

Brain regions that are selectively activated by the hierarchical acoustic features represented in each layer of the supervised classification model. Panels (A–D) represent the first four layers in the supervised classification model.