Auditory and vibrotactile interactions in perception of timbre acoustic features

Abstract

Recently, there has been increasing interest in developing auditory-to-vibrotactile sensory devices. However, the potential of these technologies is constrained by our limited understanding of which features of complex sounds can be perceived through vibrations. The present study aimed to investigate the vibrotactile perception of acoustic features related to timbre, an essential component to identify environmental, speech and musical sounds. Discrimination thresholds were measured for six features: three spectral (number of harmonics, harmonic roll-off ratio, even-harmonic attenuation) and three temporal (attack time, amplitude modulation depth and amplitude modulation frequency) using auditory, vibrotactile and combined auditory + vibrotactile stimulation in 31 adult humans with normal tactile and auditory sensitivity. Result revealed that all spectral and temporal features can be reliably discriminated via vibrotactile stimulation only. However, for spectral features, vibrotactile thresholds were significantly higher (i.e., worse) than auditory thresholds whereas, for temporal features, only vibrotactile amplitude modulation frequency was significantly higher. With simultaneous auditory and tactile presentation, thresholds significantly improved for attack time and amplitude modulation depth, but not for any of the spectral acoustic features. These results suggest that vibrotactile temporal cues have a more straightforward potential for assisting auditory perception, while vibrotactile spectral cues may require specialized signal processing schemes.

Keywords: Assistive technology; Audition; Multimodal perception; Sensory substitution; Timbre; Vibrotactile.

Fig. 1

Experimental setup and apparatus used for the experiments.

Fig. 2

Visual representation of baseline and possible contrast stimuli for each of the six acoustic features.

Fig. 3

Discrimination thresholds and multimodal gain for spectral acoustic features. The top plots (a-c) show the discrimination thresholds as boxplots, individual datapoints, and distributions for auditory (A) in blue, auditory + vibrotactile (A + VT) in yellow, and vibrotactile (VT) in red stimulation conditions. Lower thresholds indicate better discrimination performance. The bottom plots (d-f) show each participant’s multimodal gain, calculated as their auditory thresholds minus their auditory + vibrotactile thresholds (A minus A + VT), in ascending order. Negative values (in red) represent a lower performance in the multimodal condition, while positive values (in green) represent better multimodal performance. The distributions on the right show the mean multimodal gain as the dotted line, with the error bar representing the 95% CI. *p <.05. **p <.01. ***p <.001.

Fig. 4

Discrimination thresholds and multimodal gain for temporal acoustic features. The top plots (a-c) show the discrimination thresholds as boxplots, individual datapoints, and distributions for auditory (A) in blue, auditory + vibrotactile (A + VT) in yellow, and vibrotactile (VT) in red stimulation conditions. Lower thresholds indicate better discrimination performance. The bottom plots (d-f) show each participant’s multimodal gain, calculated as their auditory thresholds minus their auditory + vibrotactile thresholds (A minus A + VT), in ascending order. Negative values (in red) represent a lower performance in the multimodal condition, while positive values (in green) represent better multimodal performance. The distributions on the right show the mean multimodal gain as the dotted line, with the error bar representing the 95% CI. *p <.05. **p <.01. ***p <.001.