Learning to perceive with a visuo-auditory substitution system: localisation and object recognition with 'the vOICe'

Abstract

We investigated to what extent participants can acquire the mastery of an auditory-substitution-of-vision device ('The vOICe') using dynamic tasks in a three-dimensional environment. After extensive training, participants took part in four experiments. In the first experiment we explored locomotion and localisation abilities. Participants, blindfolded and equipped with the device, had to localise a target by moving a hand-held camera, walk towards the target, and point at it. In the second experiment, we studied the localisation ability in a constrained pointing task. In the third experiment we explored participants' ability to recognise natural objects via their auditory rendering. In the fourth experiment we tested the ability of participants to discriminate objects belonging to the same category. We analysed participants' performance from both an objective and a subjective point of view. The results showed that participants, through sensorimotor interactions with the perceptual scene while using the hand-held camera, were able to make use of the auditory stimulation to obtain the information necessary for locomotor guidance, localisation, and pointing, as well as for object recognition. Furthermore, analysis from a subjective perspective yielded insights into participants' qualitative experience and into the strategies they used to master the device, and thus to pass from a kind of deductive reasoning to a form of immediate apprehension of what is being perceived.