Spectral peak picking improves tactile speech perception

Abstract

Individual differences in speech perception often arise from disparities in access to acoustic-phonetic cues, particularly among those with hearing loss. Haptic hearing aids, which convey speech information through the sense of touch, offer a complementary pathway to improve speech understanding. However, effectively transmitting critical speech features through vibrotactile stimulation remains challenging.To address this challenge, we introduce a tactile spectral peak picking (tSPP) approach, integrated into a vocoder-based audio-to-tactile conversion algorithm to enhance vibrotactile phoneme discrimination. The tactile vocoder decomposes audio into eight frequency bands, with tSPP selectively transmitting only the most energetic bands to emphasise dominant spectral features.Tactile phoneme discrimination on the wrist was tested in 26 participants using either the tactile vocoder alone or with the tSPP algorithm selecting one, two, or four peaks. Discrimination improved significantly when one, two, or four peaks were selected relative to the vocoder alone, with the greatest benefits observed for one- and two-peak tSPP (average improvement: 7.5%).These findings demonstrate that selective enhancement of spectrally salient features can improve tactile speech perception. The algorithm is suitable for real-time use in wearable sensory substitution devices and could aid the development of effective haptic hearing aids.

Keywords: Audio-tactile; Cochlear implant; Hearing aid; Multisensory; Speech reading; Tactile aid.

Fig. 1

Phoneme discrimination performance for different numbers of peaks picked with tSPP. Results are shown separately for consonants and vowels and for the male (left panel) and female (right panel) talkers. Each box plot displays the median (horizontal line), upper and lower quartiles (box edges), and outliers (unfilled circles; defined as values more than 1.5 times the interquartile range). Whiskers indicate the full non-outlier range. Chance-level performance is marked by a dashed grey line.

Fig. 2

Performance for different consonant (left panel) and vowels (right panel) subgroups, under each tSPP condition. The box plots follow the format of Fig. 1.

Fig. 3

Spectrograms showing the audio (left panel) and the tactile envelopes with either one- or two-peak tSPP (central and right panels, respectively) for the voiced fricative consonant /z/, spoken by the female talker. The frequency band limits used for the tactile vocoder in the current experiment are marked with light blue dashed lines. The audio spectrogram sample rate was 22.05 kHz, and a 30-ms Hamming window with a hop size of 1 sample was used. The sample rate for the tactile spectrograms was 500 Hz, with no windowing applied. For the input audio, intensity is shown in decibels relative to the maximum magnitude of the short-time Fourier transform. For the tactile envelopes, intensity is shown in decibels relative to the maximum envelope amplitude.

Fig. 4

Block diagram showing the tactile vocoder signal-processing chain used to convert sound into vibration.

Fig. 5

3D renders of the EHS Research Group haptic stimulation rig. Left: the rig without the participant’s arm. Right: close-up view with the participant’s arm resting on the blue foam cushion and the shaker probe contacting the dorsal wrist. Image reproduced from Fletcher et al. with permission of the authors.