Studying human behavior with virtual reality: The Unity Experiment Framework

Abstract

Virtual reality (VR) systems offer a powerful tool for human behavior research. The ability to create three-dimensional visual scenes and to measure responses to the visual stimuli enables the behavioral researcher to test hypotheses in a manner and scale that were previously unfeasible. For example, a researcher wanting to understand interceptive timing behavior might wish to violate Newtonian mechanics so that objects can move in novel 3-D trajectories. The same researcher might wish to collect such data with hundreds of participants outside the laboratory, and the use of a VR headset makes this a realistic proposition. The difficulty facing the researcher is that sophisticated 3-D graphics engines (e.g., Unity) have been created for game designers rather than behavioral scientists. To overcome this barrier, we have created a set of tools and programming syntaxes that allow logical encoding of the common experimental features required by the behavioral scientist. The Unity Experiment Framework (UXF) allows researchers to readily implement several forms of data collection and provides them with the ability to easily modify independent variables. UXF does not offer any stimulus presentation features, so the full power of the Unity game engine can be exploited. We use a case study experiment, measuring postural sway in response to an oscillating virtual room, to show that UXF can replicate and advance upon behavioral research paradigms. We show that UXF can simplify and speed up the development of VR experiments created in commercial gaming software and facilitate the efficient acquisition of large quantities of behavioral research data.

Keywords: Behavior; Experiment; Software; Toolkit; Unity; Virtual reality.

Fig. 1

Structure of typical human behavior experiments, in the session–block–trial model. Many experiments comprise multiple repetitions of trials. Between trials, only minor changes are made. A substantial change of content in the trial is often described as creating a new block. A single iteration of a task by a participant is called a session.

Fig. 2

The UXF settings system. Independent variables that are changed as a means to iterate the design of an experiment, or to specify the experimental manipulation itself, can be written in a human-readable .json file. Settings can also be programmatically accessed or created at the trial, block, or session level. When a setting has not been specified, the request cascades up, so that the next level above is searched. This allows for both “gross” (e.g., to a whole session) and “fine” (e.g., to a single trial) storage of parameters within the same system.

Fig. 3

Screenshot of the experimenter user interface.

Fig. 4

Structure of a typical task developed with UXF. The left panel shows the functionality present in UXF, with the functionality a researcher is expected to implement shown in the right panel. The framework features several “events” (shown in red), which are invoked at different stages during the experiment; these allow developers to easily add behaviors that occur at specific times—for example, presenting a stimulus at the start of a trial.

Fig. 5

Experiment in the cloud. A piece of software developed with UXF can be deployed to an internet-connected device. Researchers can modify the experiment settings to test different experimental manipulations over time, which are downloaded from the web by the client device upon running a UXF experiment. As the participant partakes in the experiment, stimuli are presented, and the participant’s movements are recorded in the form of behaviors/responses or continuous measurement of such parameters as hand position. The results are automatically and securely streamed to a server on the internet, from which the researcher can periodically retrieve the data.

Fig. 6

Screenshot from inside the virtual room. Arrows indicate the three axes as well as the origin. The red fixation cross is shown on the wall.

Fig. 7

Head path length (where higher values indicate worse postural stability) as a function of vision condition. The two conditions were “normal” (static virtual room) and “oscillating” (oscillating virtual room). Postural stability was indexed by the path length of head movement, in meters (measured over a 10-s period). Adults showed a significantly different path length overall, as compared to children (shorter, indicating greater stability). Error bars represent ± 1 SEM.