A Novel Brain-Computer Interface Virtual Environment for Neurofeedback During Functional MRI

Abstract

Virtual environments (VEs), in the recent years, have become more prevalent in neuroscience. These VEs can offer great flexibility, replicability, and control over the presented stimuli in an immersive setting. With recent developments, it has become feasible to achieve higher-quality visuals and VEs at a reasonable investment. Our aim in this project was to develop and implement a novel real-time functional magnetic resonance imaging (rt-fMRI)-based neurofeedback (NF) training paradigm, taking into account new technological advances that allow us to integrate complex stimuli into a visually updated and engaging VE. We built upon and developed a first-person shooter in which the dynamic change of the VE was the feedback variable in the brain-computer interface (BCI). We designed a study to assess the feasibility of the BCI in creating an immersive VE for NF training. In a randomized single-blinded fMRI-based NF-training session, 24 participants were randomly allocated into one of two groups: active and reduced contingency NF. All participants completed three runs of the shooter-game VE lasting 10 min each. Brain activity in a supplementary motor area region of interest regulated the possible movement speed of the player's avatar and thus increased the reward probability. The gaming performance revealed that the participants were able to actively engage in game tasks and improve across sessions. All 24 participants reported being able to successfully employ NF strategies during the training while performing in-game tasks with significantly higher perceived NF control ratings in the NF group. Spectral analysis showed significant differential effects on brain activity between the groups. Connectivity analysis revealed significant differences, showing a lowered connectivity in the NF group compared to the reduced contingency-NF group. The self-assessment manikin ratings showed an increase in arousal in both groups but failed significance. Arousal has been linked to presence, or feelings of immersion, supporting the VE's objective. Long paradigms, such as NF in MRI settings, can lead to mental fatigue; therefore, VEs can help overcome such limitations. The rewarding achievements from gaming targets can lead to implicit learning of self-regulation and may broaden the scope of NF applications.

Keywords: brain–computer interface; methodology development; neurofeedback (NF); real-time fMRI (rtfMRI); self-regulation; virtual environment (VE).

FIGURE 1

(A) An overview of current methods used as NF modalities (Bray et al., 2007; Rota et al., 2009; Ruiz et al., 2013). These images have been modified and reimagined from the original and grouped together to concisely show the general NF visualizations. (B) Social reward via NF paradigm (Mathiak et al., 2015). This image has been converted to a gray-scale color scheme from the original. (C) An overview picture of our current implementation of the virtual environment as an NF modality. Original environment modified from Epic Games’ project titled “Shooter Game.”

FIGURE 2

Modified version of the “Shooter Game” using visibility as the NF modality. (A) The maximum luminosity condition and running crosshair bars are illustrated, whereas in (B) the luminosity is set to minimum, and the crosshair bars while the player character is stationary are illustrated.

FIGURE 3

The modified SAM rating scales as administered to the participants in the scanner. The participants could select directly one of the options on the four input buttons. The top scale is the arousal rating. The question above it asks, “How emotionally excited are you?” The options range from left “not at all” to right “very.” The bottom scale is the valence rating. The question above it asks, “How do you feel?” The options range from left “very bad” to right “very good.”

FIGURE 4

Experimental design of the gaming sessions and SAM ratings (Bradley and Lang, 1994) within the scanner.

FIGURE 5

Gaming performance over three gaming sessions shows a learning curve (Ritter and Schooler, 2001). Participants improve over time while still performing NF-based regulation via cognitive strategies.

FIGURE 6

Imagery strategies for NF employed by the participants.

FIGURE 7

(Left) Participants reporting if they perceived control over the NF modality. (Right) Amount of perceived control over neural regulation. ^∗p < 0.01.

FIGURE 8

(Left) PANAS negative affect decreased on a trend level for both groups after the NF-training. (Right) A trend to a less negative and a more positive affect change in the NF group emerged, but the comparisons failed significance.

FIGURE 9

(Left) SAM ratings showing a trend toward more arousal after the VE-NF in both groups but failing significance. (Right) SAM valence ratings did not reveal a difference between premeasurement and postmeasurement. Note that the SAM was simplified to a 4-point Likert scale.

FIGURE 10

Frequency plot showing the contribution of different frequencies in the BOLD signal from the ROI during the NF training. The NF group tends to emphasize different frequency ranges during regulation compared to the rc-NF group.

FIGURE 11

(Top) Averaged BOLD signal for the NF condition throughout the time course. The BOLD signal from our ROI has been processed using the same pipeline as in the rt-fMRI toolbox. (Bottom) Averaged BOLD signal for the rc-NF condition throughout the time course. Fitting applied a fourth-degree polynomial.

FIGURE 12

ROI-to-ROI analysis reveals connectivity differences between the two learning groups with both hippocampal regions (left and center panel). In the seed-to-voxel analysis, selective effects at the hippocampus are suggested, matching the involvement with the VE navigation task (voxel-wise threshold p < 0.001; right panel).

FIGURE 13

Average framewise displacement with 95% confidence interval in the NF and the rc-NF groups. Head motion seems to be rather lower than in previous fMRI studies on VEs or even non-motor tasks (Mathiak and Weber, 2006; Osterbauer et al., 2006).