A comparison of natural user interface and graphical user interface for narrative in HMD-based augmented reality

Abstract

Over the years, the various mediums available for storytelling have progressively expanded, from spoken to written word, then to film, and now to Virtual Reality (VR) and Augmented Reality (AR). In 2016, the cutting-edge Head-Mounted Display (HMD) AR Microsoft HoloLens was released. However, though it has been several years, the quality of the user experience with narration using HMD-based AR technology has been rarely discussed. The present study explored interactive narrative in HMD-based AR regarding different user interfaces and their influence on users' presence, narrative engagement and reflection. Inspired by an existing exhibition at the National Holocaust Centre and Museum in the U.K., a HoloLens narrative application, entitled The AR Journey, was developed by the authors using two different interaction methods, Natural User Interface (NUI) and Graphical User Interface (GUI), which were used to perform an empirical study. As revealed from the results of the between-subject design experiment, NUI exhibited statistically significant advantages in creating presence for users without 3D Role Playing Game (RPG) experience, and GUI was superior in creating presence and increasing narrative engagement for users with 3D RPG experience. As indicated by the results of the interviews, the overall narrative experience in HMD-based AR was acceptable, and the branching narrative design was engaging. However, HoloLens hardware issues, as well as virtuality and reality mismatch, adversely affected user experience. Design guidelines were proposed according to the qualitative results.

Supplementary information: The online version contains supplementary material available at 10.1007/s11042-021-11723-0.

Keywords: Augmented Reality (AR); Game experience; Head Mounted Display (HMD); HoloLens; Interactive narrative; Narrative engagement; Natural User Interface (NUI); Presence.

Fig. 1

Flow diagram of narrative structure

Fig. 2

Layout of the props (Left), concept image for the AR journey in living room (Right)

Fig. 3

Available interaction methods in HoloLens (Orange line represents the GUI design, and blue line represents the NUI design)

Fig. 4

Screen capture of different interface (1 refers to decision point interface in GUI, 2 is prop interaction in GUI, 3 is decision point interface in NUI, 4 is prop interaction in NUI)

Fig. 5

Motion capture for scene 1(left), concept composition of mocap animation and living room (right)

Fig. 6

Using faceshift to perform facial expression capture

Fig. 7

Screen captures of The AR Journey in NHCM

Fig. 8

Experimental procedure, N_RPG refers to participants with RPG experience before, N_none refers to participants without any RPG experience

Fig. 9

Participant was taking part in the experiment in China (left); the hologram mixed into the real-world space participants can be observed with HoloLens (right)

Fig. 10

Interaction plot of interaction strategy and pre-existing RPG experience for presence

Fig. 11

Interaction plot of interaction strategy and pre-existing RPG experience for feelings of control/involvement

Fig. 12

Interaction plot of interaction strategy and pre-existing RPG experience for feelings of natural interaction

Fig. 13

Interaction plot of interaction strategy and pre-existing RPG experience for narrative engagement

Fig. 14

Interaction plot of interaction strategy and pre-existing RPG experience for reflection

Fig. 15

Results for experience of sensation, interaction, interactive narrative design and overall feelings

Fig. 16

Codes of Question 2. Horizontal axis shows percentage of participants who mentioned these in their responses to the total number of all participants

Fig. 17

Codes of Question 4. Horizontal axis shows percentage of participants who mentioned these in their responses to the number of participants in their own group

Fig. 18

Codes of Question 4. Horizontal axis shows percentage of participants who mentioned these in their responses to the number of participants in their own group

Fig. 19

Codes of Question 6. Horizontal axis shows percentage of participants who mentioned these in their responses to the total number of all participants