Timbral brightness perception investigated through multimodal interference

Abstract

Brightness is among the most studied aspects of timbre perception. Psychoacoustically, sounds described as "bright" versus "dark" typically exhibit a high versus low frequency emphasis in the spectrum. However, relatively little is known about the neurocognitive mechanisms that facilitate these metaphors we listen with. Do they originate in universal magnitude representations common to more than one sensory modality? Triangulating three different interaction paradigms, we investigated using speeded classification whether intramodal, crossmodal, and amodal interference occurs when timbral brightness, as modeled by the centroid of the spectral envelope, and pitch height/visual brightness/numerical value processing are semantically congruent and incongruent. In four online experiments varying in priming strategy, onset timing, and response deadline, 189 total participants were presented with a baseline stimulus (a pitch, gray square, or numeral) then asked to quickly identify a target stimulus that is higher/lower, brighter/darker, or greater/less than the baseline after being primed with a bright or dark synthetic harmonic tone. Results suggest that timbral brightness modulates the perception of pitch and possibly visual brightness, but not numerical value. Semantically incongruent pitch height-timbral brightness shifts produced significantly slower reaction time (RT) and higher error compared to congruent pairs. In the visual task, incongruent pairings of gray squares and tones elicited slower RTs than congruent pairings (in two experiments). No interference was observed in the number comparison task. These findings shed light on the embodied and multimodal nature of experiencing timbre.

Keywords: Auditory brightness; Crossmodal correspondences; Stroop interference; Timbre; Visual brightness.

Fig. 1

Visual summary of links reported in prior studies and those explored in the present study

Fig. 2

Power spectrograms (upper panels) and spectra (lower panels) of the “dark” (low spectral centroid (SC)) and “bright” (high SC) tones (same pitch) used as primes in the auditory-visual and -numerical tasks, and as baselines in the pitch-timbre and timbre-pitch tasks

Fig. 3

Experimental procedure

Fig. 4

Experiment 1 choice reaction time (RT; A) and response error (B) to target pitch height in congruent and incongruent pairings with timbre shift. Experiment 4 choice RT (C) and response error (D) to target timbre in congruent and incongruent pairings with pitch shift. Error: SEM

Fig. 5

Choice reaction time (RT) to target square brightness in congruent and incongruent pairings with timbral brightness shift in Experiments 1 (A) and 3 (B). Error: SEM