Improvement of Apraxia With Augmented Reality: Influencing Pantomime of Tool Use via Holographic Cues

Abstract

Background: Defective pantomime of tool use is a hall mark of limb apraxia. Contextual information has been demonstrated to improve tool use performance. Further, knowledge about the potential impact of technological aids such as augmented reality for patients with limb apraxia is still scarce. Objective: Since augmented reality offers a new way to provide contextual information, we applied it to pantomime of tool use. We hypothesize that the disturbed movement execution can be mitigated by holographic stimulation. If visual stimuli facilitate the access to the appropriate motor program in patients with apraxia, their performance should improve with increased saliency, i.e., should be better when supported by dynamic and holographic cues vs. static and screen-based cues. Methods: Twenty one stroke patients and 23 healthy control subjects were randomized to mime the use of five objects, presented in two Environments (Screen vs. Head Mounted Display, HMD) and two Modes (Static vs. Dynamic) resulting in four conditions (Screen^Stat, Screen^Dyn, HMD^Stat, HMD^Dyn), followed by a real tool demonstration. Pantomiming was analyzed by a scoring system using video recordings. Additionally, the sense of presence was assessed using a questionnaire. Results: Healthy control participants performed close to ceiling and significantly better than patients. Patients achieved significantly higher scores with holographic or dynamic cues. Remarkably, when their performance was supported by animated holographic cues (e.g., striking hammer), it did not differ significantly from real tool demonstration. As the sense of presence increases with animated holograms, so does the pantomiming. Conclusion: Patients' performance improved with visual stimuli of increasing saliency. Future assistive technology could be implemented upon this knowledge and thus, positively impact the rehabilitation process and a patient's autonomy.

Keywords: apraxia; hologram; pantomime of tool use; sense of presence; stroke; virtual reality; visual cues.

Figure 1

Patients' flow through the study. All participants had to perform the pantomime task twice (Static/Dynamic) in each Environment (HMD/Screen), followed by the real tool condition. The washout time was set to >24 h and did not include any additional tasks. Three participants only completed Day 1 and were excluded from further analyses.

Figure 2

Third person perspective of the experimental setup. (A) Screen condition, (B) HMD condition, (C) Real Tool condition; and (D) the first-person perspective of the five objects depicted as screen-based images. Only the tools and not the target items were shown (i.e., the hammer, but not a nail).

Figure 3

Trajectories of hand movements in patient P13 attempting to pantomime the typical use of a hammer during the different experimental conditions is illustrated: Screen^Stat (blue line), Screen^Dyn (blue dotted line), HMD^Stat (red line), HMD^Dyn (red dotted line), real tool (gray line). The complete trajectory along the z-Axis in (mm) during the third of three trials is always shown.

Figure 4

(A,B) Results of the pantomime performance of the control group (left) and stroke group (right); Mean and 95% confidence interval values for the interactions of Environment (HMD/Screen) and Mode (static/dynamic) in (A) Production scale and (B) Interaction scale are reported.

Figure 5

Conditional (DYN), environmental (HMD) and combined (HOLO) effects of individual patients displayed in (A) Production Scale and (B) Interaction Scale.