Using immersive virtual reality to recreate the synaesthetic experience

Abstract

Synaesthesia is a condition where people experience unusual sensory or cognitive sensations in response to apparently unrelated stimuli. This paper presents two experiments which aimed to examine whether Virtual Reality (VR) technology can be used to recreate the synaesthetic experience. There is a lack of research in this area, with most studies focussing primarily on synaesthetic colors. Experiment 1 aimed to build on previous research by using not only a traditional color-picker but also VR to capture a more nuanced picture of synaesthetic perception. A multiple case study design was used to examine the experiences of six participants in detail. Data gathering took place via Zoom. During the initial data-gathering session, participants used a color-picker to provide grapheme-color associations. After this session, some of the participants' synaesthetic experiences were recreated using a VR-by-proxy approach. Results indicated that VR is capable of capturing elements of synaesthetic perception that other methods have been unable to, such as texture, small degrees of movement, and 3D structure. Experiment 2 expanded upon these findings by moving beyond the VR-by-proxy approach and asking three participants to recreate their own audiovisual synaesthetic associations in the VR environment. Inductive Thematic Analysis was used to analyze the results of this experiment. The potential of expanding this technique to other forms of perceptual diversity is discussed.

Keywords: cross-modal correspondences; perception; synaesthesia; virtual reality.

Figure 1.

Summary of the stimuli presented to each participant.

Figure 2.

Flowchart illustrating the experimental procedure.

Figure 3.

Grapheme-color associations for numbers.

Figure 4.

Grapheme-color associations for days.

Figure 5.

Grapheme-color associations for months.

Figure 6.

Example screenshots of OpenBrush (2016) videos created by the researcher, using participants’ descriptions.

Figure 7.

Participant 1 drawings in response to experimental stimuli.

Figure 8.

Participant 2 drawings in response to experimental stimuli.

Figure 9.

Participant 3 drawings in response to experimental stimuli.

Figure 10.

Top image is a drawing produced by Participant 1 in response to the cars passing stimulus and the bottom image represents drawing produced by Participant 2 in response to the phone ringing stimulus. Several symbols were selected to represent the movement captured in the Virtual Reality (VR) environment: squiggle movement referring to serrated/wave-like movements; directional movement referring to movement in one specific direction; smoke referring to smoke-like animated brush, which was often used by the participants; and sparkle—animated sparkle brush which was fading in and out. Videos of the drawings can be found in online supplementary materials.

Figure 11.

This figure represents video captures of their drawings for all stimuli for both the original experiment and the retest.

Figure 11.

This figure represents video captures of their drawings for all stimuli for both the original experiment and the retest.

Figure 12.

This figure represents spatial orientation of drawings in relation to the participant. Participant 1 (top) viewed things as if on a screen and felt constrained by their visual field, which made them reluctant to explore the three-dimensional space of the virtual world; Participant 2 (middle) drew things relative to their body and discussed moving through and standing on objects in the three-dimensional space; Participant 3 (bottom) again drew objects in relation to their body, but typically beside, behind or in front of them.