An investigation into the effectiveness of using acoustic touch to assist people who are blind

Abstract

Wearable smart glasses are an emerging technology gaining popularity in the assistive technologies industry. Smart glasses aids typically leverage computer vision and other sensory information to translate the wearer's surrounding into computer-synthesized speech. In this work, we explored the potential of a new technique known as "acoustic touch" to provide a wearable spatial audio solution for assisting people who are blind in finding objects. In contrast to traditional systems, this technique uses smart glasses to sonify objects into distinct sound auditory icons when the object enters the device's field of view. We developed a wearable Foveated Audio Device to study the efficacy and usability of using acoustic touch to search, memorize, and reach items. Our evaluation study involved 14 participants, 7 blind or low-visioned and 7 blindfolded sighted (as a control group) participants. We compared the wearable device to two idealized conditions, a verbal clock face description and a sequential audio presentation through external speakers. We found that the wearable device can effectively aid the recognition and reaching of an object. We also observed that the device does not significantly increase the user's cognitive workload. These promising results suggest that acoustic touch can provide a wearable and effective method of sensory augmentation.

Fig 1

(A) The study set-up for the seated task of the study. The participants were asked to use the information provided (either using the provided wearable device or an external source) to identify and memorize the item locations on the table. The participant would then be asked to search and reach for a specific item on the table using the memorized map (without any aids). (B) The standing task requires the participant to use the provided wearable device to search and reach for a target item situated among multiple distractor items (bottles).

Fig 2

(A) The NReal AR glasses worn by a participant. The grey spheres are the reflective markers that tracked the participant’s head movement through motion capture. The participant’s face has been blurred for privacy. (B) The household items used during the study. The items (from left to right) are the bowl, book, cup, and bottle. (C) The Sony XB13 Bluetooth speakers and the user response button were used during the study.

Fig 3. A demonstration of the NReal glasses, OPPO android smartphone, and the sonification app being used to detect a cup and bottle within the user’s FOV.

Fig 4. A block diagram outlining the connections between the technical components and the study’s users (researchers and participants).

The arrows represent the flow of information with the text specifying the type of information.

Fig 5. An outline of the single trial protocol for the seated task.

The red circle indicates the non-automated aspect where the researcher or participant chooses to advance the trial through a key or button press.

Fig 6

(A) depicts the hand, head, and item tracking through motion capture in this study. (B) Plots of the hand trajectory (z-x plane, overhead view) for every FAD trial by the participants. The blue dots represent the hand’s starting position with an offset always initialized at (0,0). The red dots represent reached item position. Position 1–5 refers to reaching positions from the left of the user to the right.

Fig 7

(A) The average with standard error bars of the seated task success rate sorted by each condition for both the BLV and blindfolded sighted participants. (B) The average with standard error bars of each item’s detection rate when using the FAD to search for the items on the table.

Fig 8

(A) The average with standard error bars of the seated task success rate sorted by each item and condition for both the BLV and blindfolded sighted participants. (B) The average with standard error bars of the number of attempts to reach for the target item during the standing task.

Fig 9. The box plots show the hand reaching speed, trajectory-to-optimal ratio, and the reaching time for the BLV and blindfolded sighted participant groups.

Fig 10. The box plots for the total head rotation and head rotational speed during the scanning-for-items stage of the FAD condition are shown for the seated task.

It should be noted that the item could be placed directly on either side of the participant; a single head sweep to search the table would be 180° rotation.

Fig 11. The average values with standard error bars are shown for each NASA-TLX category and the total (RAW-TLX) workload value for the BLV and Sighted participant groups.

The scoring categories are Mental, Physical (Phys), Temporal (Temp), Performance (Per), Effort (Eff), Frustration (Frus), and Total Raw-TLX score. B.S. refers to the Blindfolded Sighted participants.