Audio-visual perception of 3D cinematography: an fMRI study using condition-based and computation-based analyses

Abstract

The use of naturalistic stimuli to probe sensory functions in the human brain is gaining increasing interest. Previous imaging studies examined brain activity associated with the processing of cinematographic material using both standard "condition-based" designs, as well as "computational" methods based on the extraction of time-varying features of the stimuli (e.g. motion). Here, we exploited both approaches to investigate the neural correlates of complex visual and auditory spatial signals in cinematography. In the first experiment, the participants watched a piece of a commercial movie presented in four blocked conditions: 3D vision with surround sounds (3D-Surround), 3D with monaural sound (3D-Mono), 2D-Surround, and 2D-Mono. In the second experiment, they watched two different segments of the movie both presented continuously in 3D-Surround. The blocked presentation served for standard condition-based analyses, while all datasets were submitted to computation-based analyses. The latter assessed where activity co-varied with visual disparity signals and the complexity of auditory multi-sources signals. The blocked analyses associated 3D viewing with the activation of the dorsal and lateral occipital cortex and superior parietal lobule, while the surround sounds activated the superior and middle temporal gyri (S/MTG). The computation-based analyses revealed the effects of absolute disparity in dorsal occipital and posterior parietal cortices and of disparity gradients in the posterior middle temporal gyrus plus the inferior frontal gyrus. The complexity of the surround sounds was associated with activity in specific sub-regions of S/MTG, even after accounting for changes of sound intensity. These results demonstrate that the processing of naturalistic audio-visual signals entails an extensive set of visual and auditory areas, and that computation-based analyses can track the contribution of complex spatial aspects characterizing such life-like stimuli.

Figure 1. The multi-speakers system and the computation of absolute disparity and disparity gradient.

A. Schematic illustration of the multi-speakers system used for sound-surround stimulation in the scanner. The system utilizes five independent sound-lines: a central channel (C: comprising two speakers delivering the same signal); two front channels (F_L/F_R) and two back channels (B_L/B_R). The drawings also show the approximate position of the screen (S) and of the mirror (M) used to show the visual stimuli. B. An example of a video frame, with the two different images for the left and right eye. The left and right images were projected thought a linear polarizer and were perceived as a single 3D image using a passive eyewear. C. The corresponding map of “absolute” disparity, computed using the algorithm of HL-SIFT flow . D. The disparity “gradient” map associated with the same frame. This was computed by extracting the local intensity contrast of the absolute map, via Gaussian pyramid decomposition (see Methods section).

Figure 2. The results of condition-based analyses of Exp 1 (see Table 1 ), rendered on the brain template of SPM.

A. Activation for the main effect of “3D>2D” visual stimulation. The signal plots show the parameter estimates in V3A and LOC, separately for the 4 experimental conditions. B. Activation for the main effect of “Surround>Mono” auditory stimulation. The signal plots show the parameter estimates in the superior temporal gyrus (STG). All activations are displayed at a threshold of p-unc. = 0.001. The signal plots show the average activity within each cluster, extracted using MarsBaR . Parameter estimates are in arbitrary units, error bars are standard errors.

Figure 3. The result of computation-based analysis in Exp 2 and Exp 1 (see Table 3 ), shown on transverse and coronal sections of the SPM template, in neurological convention.

A. Activations associated with the absolute disparity index, with significant effects in V3A (Exp 2 and Exp 1), plus the pSPL and the fusiform gyrus in Exp 2 only. Activations are displayed at a threshold of p-unc. = 0.001; colorbars show t-values. B. Activations associated with the disparity gradient index, showing consistent effects in the posterior middle temporal gyrus bilaterally (pMTG) and the left inferior frontal gyrus (IFG) both in Exp 2 and Exp 1. Activations are displayed at a threshold of p-unc. = 0.001; colorbars show t-values. C. Activations associated with the auditory “complexity” index that, in Exp 2, included the planum temporale (PT) and TE sub-regions in the auditory cortex (cf. Table 4; and see Fig. 4, for a detailed view). These effects in auditory cortex did not replicate in Exp 1, after accounting for the block effect of “surround vs. mono” presentation (cf. sections on the right). Activations are displayed at a threshold of p-FWE-corr. = 0.05; colorbars show F-values. D. Activations associated with the auditory “intensity” index, revealing effects in STG and STS, plus some influence also in the occipital visual cortex (cf. results). Activations are displayed at a threshold of p-FWE-corr. = 0.05; colorbars show F-values.

Figure 4. Detailed view of Heschl's gyrus (HG), with the layout of the different auditory effects in the auditory cortex (AC) and the planum temporale (PT).

This shows that the blocked effect of surround sounds (in Exp 1) and time-varying effects of auditory intensity (in Exp 2) activated all TE sub-regions of AC plus the PT (in yellow and in violet). By contrast, the effect of auditory “complexity” activated the PT and only the most lateral part of AC (in yellow and in orange), without affecting area TE1.1. The broken lines show the borders between the TE sub-regions (TE1.0, TE1.1, and TE1.2) estimated using the SPM anatomy toolbox . Activations are displayed on axial section (z = 2).