A self-training program for sensory substitution devices

Abstract

Sensory Substitution Devices (SSDs) convey visual information through audition or touch, targeting blind and visually impaired individuals. One bottleneck towards adopting SSDs in everyday life by blind users, is the constant dependency on sighted instructors throughout the learning process. Here, we present a proof-of-concept for the efficacy of an online self-training program developed for learning the basics of the EyeMusic visual-to-auditory SSD tested on sighted blindfolded participants. Additionally, aiming to identify the best training strategy to be later re-adapted for the blind, we compared multisensory vs. unisensory as well as perceptual vs. descriptive feedback approaches. To these aims, sighted participants performed identical SSD-stimuli identification tests before and after ~75 minutes of self-training on the EyeMusic algorithm. Participants were divided into five groups, differing by the feedback delivered during training: auditory-descriptive, audio-visual textual description, audio-visual perceptual simultaneous and interleaved, and a control group which had no training. At baseline, before any EyeMusic training, participants SSD objects' identification was significantly above chance, highlighting the algorithm's intuitiveness. Furthermore, self-training led to a significant improvement in accuracy between pre- and post-training tests in each of the four feedback groups versus control, though no significant difference emerged among those groups. Nonetheless, significant correlations between individual post-training success rates and various learning measures acquired during training, suggest a trend for an advantage of multisensory vs. unisensory feedback strategies, while no trend emerged for perceptual vs. descriptive strategies. The success at baseline strengthens the conclusion that cross-modal correspondences facilitate learning, given SSD algorithms are based on such correspondences. Additionally, and crucially, the results highlight the feasibility of self-training for the first stages of SSD learning, and suggest that for these initial stages, unisensory training, easily implemented also for blind and visually impaired individuals, may suffice. Together, these findings will potentially boost the use of SSDs for rehabilitation.

Fig 1. EyeMusic description.

The EyeMusic visual-to-auditory SSD transforms visual information into auditory soundscapes. X-axis information is conveyed through time, such that information on the left side of the image is heard before the information on the right. Y-axis information is conveyed through pitch manipulations on the pentatonic scale, such that objects ‘features positioned in the higher portions of the image are sonified with a higher pitch than lower features. Colors are conveyed through timbre variations using different musical instruments. In the current experiment, we used the colors red (piano), white (choir) and blue (trumpet), while silence conveyed black. The orange box, sweeps from left to right, sonifying one column at a time.

Fig 2. Experimental flow.

The experiment included 5 groups of sighted participants, 4 experimental groups, and 1 control group. All participants performed a baseline SSD identification test, and repeated the same test after ~75 minutes. Between tests, the 4 experimental groups participated in a self-learning online training program consisting of 9 step-by-step lessons of increasing difficulty guiding them through the basic principles of the EyeMusic. The feedback method deployed during training to teach the participants to interpret the auditory stimuli of the EyeMusic, varied among groups: 1 Auditory only unisensory group receiving an auditory description of the stimuli after each EyeMusic stimulus; and 3 Audio-visual multi-sensory groups—2 groups perceiving visual images appearing either simultaneously or following the EyeMusic stimuli; 1 group receiving textual descriptions of the stimuli alongside hearing the auditory stimulus; In the control group participants were instructed to free reading from the computer during the ~75 minutes between the two SSD identification tasks.

Fig 3. Pre- and post-training accuracy in the SSD identification test for all experimental groups.

Baseline average accuracy level in the pre-training test is depicted in the bottom part of each stacked bar. Average accuracy in the post-training test is depicted in the top part of each stacked bar (shaded colors). First, when comparing accuracy in the pre-training test, no difference was observed between experimental groups (Kruskal-Wallis p-value = 0.2). Pooling the baseline measurement amongst all participants from all experimental conditions (43% ± 12%, pink bar) was significantly higher than a chance level of 25% (two-sample t-test, unequal variance, p< 0.00001, asterisk on top of the bar). Importantly, post-training accuracy rate in each of the four training groups was significantly higher than their accuracy in the pre-training SSD identification test (Wilcoxon sign-rank, auditory only (unisensory) p-value = 0.004; interleaved audio-visual (multisensory) p-value = 0.002; simultaneous audio-visual (multisensory) p-value = 0.002; simultaneous textual description (multisensory) p-value = 0.0098). This was not the case in the control group (Wilcoxon sign-rank, p-value = 0.9) (asterisks on top of the stacked bars). Additionally, when calculating the improvement-in-accuracy index as the difference in accuracy between pre- and post-training tests (shaded bar graphs), a significant effect emerged among experimental groups (Kruskal-Wallis p-value = 0.006). Post-hoc Wilcoxon rank-sum analysis revealed that this was driven by a significant difference between the control condition and all four training conditions (auditory only (unisensory) vs. control p-value = 0.006, interleaved audio-visual (multisensory) vs. control p-value = 0.001, simultaneous audio-visual (multisensory) vs. control p-value = 0.002, simultaneous textual description (multisensory) vs. control p-value = 0.03), while no other differences were significant (all p-values >0.33). Note that in all the stacked bars depicted here, error bars show the standard error.

Fig 4. Improvement in accuracy between pre- and post-training SSD identification test for perceptual and descriptive training groups.

When pooling together the improvement-in-accuracy index for all perceptual training strategies (interleaved and simultaneous audio-visual multisensory), and the improvement-in-accuracy index for the descriptive training strategies (auditory only unisensory, and audio-visual textual descriptive), no significant difference was found (rank-sum, p = 0.99). Note that the error bars show the standard error.

Fig 5. Individual accuracy in pre- and post-training SSD identification test, separated for each experimental group.

Each graph shows the success rate of a single participant in pre-training (dark bars) and post-training SSD identification tests (light bars). A. Auditory only (unisensory) training group: 8 out of 10 participants improved their success rate in the post-training test compared to their pre-training performance. B. Interleaved audio-visual (multisensory) training group: all participants improved their success rate in the post-training test compared to their pre-training performance. C. Simultaneous audio-visual (multisensory) training group: all participants improved their success rate in the post-training test compared to their pre-training performance. D. Simultaneous textual description (multisensory) training group: 9 out of 10 participants improved their success rate in the post-training test compared to their pre-training performance (note that 1 out of these 9 participants showed a very minimal improvement in the post-training test). E. Control group: 4 out of 10 participants improved their success rate in the post-training test compared to their pre-training performance (note that 2 out of these 4 participants showed a very minimal improvement in the post-training test).

Fig 6. Participants self-assessment.

Following each training lesson, and before the end-lesson quiz, participants rated in a 1–5 scale two self-assessment questions: 1) their subjectively perceived learning of the material presented in each lesson, 2) how difficult they subjectively rated each lesson. A. Learning self-assessment: The median of self-evaluation of learning, was highest for participants from the interleaved audio-visual (multisensory) group (blue), followed by participants from the auditory only (unisensory) group (green). B. Difficulty self-assessment: The median of self-evaluation of difficulty, was lowest for participants from the interleaved audio-visual (multisensory) group (blue), followed by participants from the auditory only (unisensory) group (green). Note that all error bars here represent MAD.