Higher-order Sonification of the Human Brain

Abstract

Sonification, the process of translating data into sound, has recently gained traction as a tool for both disseminating scientific findings and enabling visually impaired individuals to analyze data. Despite its potential, most current sonification methods remain limited to one-dimensional data, primarily due to the absence of practical, quantitative, and robust techniques for handling multi-dimensional datasets. We analyze structural magnetic resonance imaging (MRI) data of the human brain by integrating two- and three-point statistical measures in Fourier space: the power spectrum and bispectrum. These quantify the spatial correlations of 3D voxel intensity distributions, yielding reduced bispectra that capture higher-order interactions. To showcase the potential of the sonification approach, we focus on a reduced bispectrum configuration which applied to the OASIS-3 dataset (864 imaging sessions), yields a brain age regression model with a mean absolute error (MAE) of 4.7 years. Finally, we apply sonification to the ensemble-averaged (median) outputs of this configuration across five age groups: 40-50, 50-60, 60-70, 70-80, and 80-100 years. The auditory experience clearly reveals differentiations between these age groups, an observation further supported visually when inspecting the corresponding sheet music scores. Our results demonstrate that the information loss (e.g., normalized mean squared error) during the reconstruction of the original bispectra, specifically in configurations sensitive to brain aging, from the sonified signal is minimal. This approach allows us to encode multi-dimensional data into time-series-like arrays suitable for sonification, creating new opportunities for scientific exploration and enhancing accessibility for a broader audience.

Figure 1.

This diagram illustrates the sonification method for three-dimensional data, utilizing higher-order statistical analysis in Fourier space. The two-point statistics are represented by sticks of varying lengths, which connect pairs of voxels (in the configuration space analog), indicating their respective intensities from the MRI scan. The three-point statistics are depicted through different triangle configurations, where two fixed side lengths are considered with varying subtended angles to connect three voxels. A one-dimensional function is derived by combining these statistical measures, which can then be directly sonified.

Figure 2.

Sonification piano: notes used in this study covering the frequency range within the sensitivity of adult humans (20–30 yrs: ~16, 40 yrs: ~14, 50 yrs: ~12, 60 yrs: ~10, 70 yrs: ~8 kHz). Regular notes (${\text{C}}_i,{\text{D}}_i,{\text{E}}_i,{\text{F}}_i,{\text{G}}_i,{\text{A}}_i,{\text{B}}_i$ for different octaves $i$ ranging from 1 to 8) have been split into two columns (first two on the left) for visualisation purposes. The same has been done with quarter tones (indicated by +25) on the right. Semitones are indicated in the third column on the left. The piano convention of black keys for semitones is adopted. We consider sonification cases with and without quarter tones depending on the bispectrum configuration and the desired accuracy.

Figure 3.

On the left: age distribution for the OASIS-3 864 MRI sessions dataset reduced by Carnero-Rosell et al. with the corresponding MAE at different AGE bins according to Random Forests classification based solely on the Q019036 bispectrum configuration. On the right: corresponding age regression.

Figure 4.

Sonification of bispectra from human MRI, using an age group of 80–100 years as an example. The original data and the corresponding inverse mapping after sonification are displayed, with the bispectrum for the 40–50 age group included as a reference. The ratios and differences between the original and the inverse-mapped sonified signals are also presented.

Figure 5.

Scores corresponding to (top:) the age group of 80 to 100 using as the reference the group of 40 to 50 to set the range; and (bottom:) the age group of 40 to 50.

Figure 6.

Scores corresponding to the difference between the age group of 80 to 100 with the group of 40 to 50.

Figure 7.

Sonification procedure across four panels: The upper-left panel shows the original signal with each data point represented by a small circle. In this example, the difference of the reduced bispectra (configuration $k_1=0.19$ and $k_2=0.36$) for age groups: 80–100 and 40–50 is shown. In the upper-right panel, the minimum value is subtracted from the signal. The lower-left panel illustrates the signal discretization process, additionally applying a linear mapping to convert the signal into the range of MIDI values that are perceptible to adult human listeners. Finally, the lower-right panel demonstrates the effect of incorporating a nonlinear rank-ordered transformation with a power-law of exponent $f_{\text{exp}}=0.45$. The velocity (volume) of the sonified signal is represented through the size of the circles. The normalization is one.

Figure 8.

Similar sonification procedure to Figure 7, but showing relative reduced bispectra relations (configuration $k_1=0.19$ and $k_2=0.36$). The different panels show the discretized signal for the 50–60 (upper-left), 60–70 (upper-right), 70–80 (lower-left), 80–100 (lower-right) age group within the range of the reference 40–50 years group mapped to the corresponding MIDI values. The filled circles stand for the reference bispectrum.

Figure 9.

Sonification of reduced bispectra (configuration $k_1=0.19$ and $k_2=0.36$) from human brain magnetic resonance imaging for different ages. The original data and the corresponding inverse mapping after sonification are indicated for each age range.

Figure 10.

Sonification of difference between reduced bispectra (configuration $k_1=0.19$ and $k_2=0.36$) from human brain magnetic resonance imaging for different ages groups with respect to the youngest age group in our sample (40–50 yrs). Upper-left panel: $c_{\text{norm}}=0.25,f_{\text{exp}}=0.7$; upper-right panel: $c_{\text{norm}}=0.5,f_{\text{exp}}=0.35$; lower-left panel: $c_{\text{norm}}=0.75,f_{\text{exp}}=0.35$; lower-right panel: $c_{\text{norm}}=1,f_{\text{exp}}=0.45$.