Psycho-physiological assessment of a prosthetic hand sensory feedback system based on an auditory display: a preliminary study

Abstract

Background: Prosthetic hand users have to rely extensively on visual feedback, which seems to lead to a high conscious burden for the users, in order to manipulate their prosthetic devices. Indirect methods (electro-cutaneous, vibrotactile, auditory cues) have been used to convey information from the artificial limb to the amputee, but the usability and advantages of these feedback methods were explored mainly by looking at the performance results, not taking into account measurements of the user's mental effort, attention, and emotions. The main objective of this study was to explore the feasibility of using psycho-physiological measurements to assess cognitive effort when manipulating a robot hand with and without the usage of a sensory substitution system based on auditory feedback, and how these psycho-physiological recordings relate to temporal and grasping performance in a static setting.

Methods: 10 male subjects (26+/-years old), participated in this study and were asked to come for 2 consecutive days. On the first day the experiment objective, tasks, and experiment setting was explained. Then, they completed a 30 minutes guided training. On the second day each subject was tested in 3 different modalities: Auditory Feedback only control (AF), Visual Feedback only control (VF), and Audiovisual Feedback control (AVF). For each modality they were asked to perform 10 trials. At the end of each test, the subject had to answer the NASA TLX questionnaire. Also, during the test the subject's EEG, ECG, electro-dermal activity (EDA), and respiration rate were measured.

Results: The results show that a higher mental effort is needed when the subjects rely only on their vision, and that this effort seems to be reduced when auditory feedback is added to the human-machine interaction (multimodal feedback). Furthermore, better temporal performance and better grasping performance was obtained in the audiovisual modality.

Conclusions: The performance improvements when using auditory cues, along with vision (multimodal feedback), can be attributed to a reduced attentional demand during the task, which can be attributed to a visual "pop-out" or enhance effect. Also, the NASA TLX, the EEG's Alpha and Beta band, and the Heart Rate could be used to further evaluate sensory feedback systems in prosthetic applications.

Figure 1

Experiment setting. a) Subject’s perspective of the robot hand during the experiment. The task was to grasp a bottle with the prosthetic hand. b) View of the whole experiment setting. The subject used a Data Glove to control the robot hand motion and bending sensors in the robot hand fingers were used to produce the auditory feedback cues.

Figure 2

Experiment setting block diagram. The subjects controlled the robot hand’s motion using a Data Glove. The different profiles of the robot hand movements were mapped to sounds and conveyed back to the subjects. During tests, the experimenter forced one of the robot hand’s fingers to stop moving, which generated an error signal that was conveyed back to the subject as an auditory icon.

Figure 3

Temporal performance results. a) Trial Duration. It can be noted that the trial duration was lower when the auditory and the visual feedback was used simultaneously. b) Duration to detect and fix an error. An error was detected and fixed faster when the auditory display was used.

Figure 4

Finger flexure results. The finger flexure was measured as a percentage value where 0% indicated a fully open hand pose and 100% indicated a fully closed hand pose (e.g. fist). From the results it can be noted that the Auditory feedback allowed the subject to know more accurately when the bottle was completely grasped.

Figure 5

NASA TLX scores. The NASA TLX mean scores showed that, in general, the VF modality was ranked with the highest mental workload, followed very closely by the AF modality when manipulating the prosthetic device.

Figure 6

NASA TLX scores without outliers. a) Subject 1 and Subject 8 were the only ones that reported that the mental effort in the VF modality was lower than in the other 2 modalities, which were considered to be similarly difficult. b) Considering Subject 1 and Subject 8 as outliers, the results showed the same tendency, but with a significant effect between modalities.

Figure 7

EEG results. a) The EEG mean power spectra of the Alpha and Beta band of each modality for all subjects. A higher Alpha suppression was found for the VF modality. b) When considering Subject 1 and 8 as outliers, we found a significantly statistical effect between the modalities. These results points out a higher attentional demand in the VF modality.

Figure 8

ECG results. The VF modality showed the lowest deceleration of all the groups, followed very closely by the AF modality, while the AVF showed the highest deceleration. Which points out a lower mental effort and more focused attention in the AVF modality.

Figure 9

Attentional model. Sensory information is integrated, enhancing or reducing the attention to a specific task. Then, this information is processed in order to understand the meaning of the sensory input and to decide what actions should be taken. Long term memory wasn’t taken in consideration in this study.