Improving speech perception for hearing-impaired listeners using audio-to-tactile sensory substitution with multiple frequency channels

Abstract

Cochlear implants (CIs) have revolutionised treatment of hearing loss, but large populations globally cannot access them either because of disorders that prevent implantation or because they are expensive and require specialist surgery. Recent technology developments mean that haptic aids, which transmit speech through vibration, could offer a viable low-cost, non-invasive alternative. One important development is that compact haptic actuators can now deliver intense stimulation across multiple frequencies. We explored whether these multiple frequency channels can transfer spectral information to improve tactile phoneme discrimination. To convert audio to vibration, the speech amplitude envelope was extracted from one or more audio frequency bands and used to amplitude modulate one or more vibro-tactile tones delivered to a single-site on the wrist. In 26 participants with normal touch sensitivity, tactile-only phoneme discrimination was assessed with one, four, or eight frequency bands. Compared to one frequency band, performance improved by 5.9% with four frequency bands and by 8.4% with eight frequency bands. The multi-band signal-processing approach can be implemented in real-time on a compact device, and the vibro-tactile tones can be reproduced by the latest compact, low-powered actuators. This approach could therefore readily be implemented in a low-cost haptic hearing aid to deliver real-world benefits.

Figure 1

Percentage of phoneme pairs discriminated for each experimental condition, with chance performance marked by a dashed grey line. Stars show the statistical significance of differences between conditions (corrected for multiple comparisons), with more stars indicating greater significance. Error bars show the standard error of the mean (SEM).

Figure 2

Percentage of phoneme pairs discriminated for each experimental condition, with consonant and vowel pairs shown separately. Error bars show the standard error of the mean (SEM).

Figure 3

Percentage of phoneme pairs discriminated for the four-vibro-tactile-tone conditions (one or four frequency bands), grouped by phoneme contrast type. Stars show the statistical significance of differences between one and four frequency bands (corrected for multiple comparisons), with more stars indicating greater significance. Error bars show the SEM. Chance performance is marked with a dashed grey line.

Figure 4

Percentage of phoneme pairs discriminated for the eight-vibro-tactile-tone experimental conditions (one or eight frequency bands), grouped by phoneme contrast type. Stars show the statistical significance of differences between one and eight frequency bands (corrected for multiple comparisons), with more stars indicating greater significance. Error bars show the SEM. Chance performance is marked with a dashed grey line.

Figure 5

The improvement in the percentage of phoneme pairs discriminated for four or eight frequency bands compared to one frequency band, grouped by phoneme contrast type. Error bars show the SEM.

Figure 6

Spectrograms showing the input audio (left panel) and the tactile envelopes extracted using the eight-frequency-channel vocoder approach (right panel) for the phonemes æ and e (spoken by the male talker). The first and second formants of the input audio are marked. The upper two frequency channels and lowest channel are not shown for the tactile envelopes. The audio spectrogram sample rate was 22.05 kHz, with a window size of 1024 (Hann) and a hop size of 1 sample. The tactile spectrogram sample rate was 16 kHz, and no windowing was applied to the envelopes. Intensity is shown in decibels relative to the maximum magnitude of the STFT for the input audio and in decibels relative to the maximum envelope amplitude for the tactile envelopes. The spectrograms were generated using the Librosa Python library (version 0.10.0).

Figure 7

The long-term average spectrum of the male and female talker from the EHS Research Group Phoneme Corpus (based on all phonemes), with no normalisation applied.

Figure 8

A 3D rendered image of the EHS Research Group haptic stimulation rig used in the current study. The left image shows the set up with no arm in place and the shaker and probe free hanging. The right image shows a close view of the rig with the arm in place and the shaker probe contacting the wrist.