Affective Interaction with a Virtual Character Through an fNIRS Brain-Computer Interface

Abstract

Affective brain-computer interfaces (BCI) harness Neuroscience knowledge to develop affective interaction from first principles. In this article, we explore affective engagement with a virtual agent through Neurofeedback (NF). We report an experiment where subjects engage with a virtual agent by expressing positive attitudes towards her under a NF paradigm. We use for affective input the asymmetric activity in the dorsolateral prefrontal cortex (DL-PFC), which has been previously found to be related to the high-level affective-motivational dimension of approach/avoidance. The magnitude of left-asymmetric DL-PFC activity, measured using functional near infrared spectroscopy (fNIRS) and treated as a proxy for approach, is mapped onto a control mechanism for the virtual agent's facial expressions, in which action units (AUs) are activated through a neural network. We carried out an experiment with 18 subjects, which demonstrated that subjects are able to successfully engage with the virtual agent by controlling their mental disposition through NF, and that they perceived the agent's responses as realistic and consistent with their projected mental disposition. This interaction paradigm is particularly relevant in the case of affective BCI as it facilitates the volitional activation of specific areas normally not under conscious control. Overall, our contribution reconciles a model of affect derived from brain metabolic data with an ecologically valid, yet computationally controllable, virtual affective communication environment.

Keywords: affective computing; brain-computer interfaces; fNIRS; neurofeedback; virtual agents.

Figure 1

System overview. Brain signals are collected through functional near infrared spectroscopy (fNIRS) system (A) where left-most and right-most channels are processed to generate a left-asymmetry score (1). During the View epoch (2), the left-asymmetry values are used to define the Min and Max bounds (3) to be used during the Neurofeedback (NF) epoch where the real-time left-asymmetry scores (4) are normalized (B) before being used as single input (5) to the virtual agent’s facial expressions action units (AUs) and body action parameters under the neural network’s control (C).

Figure 2

Progression of the agent’s behaviors according to its level of engagement.

Figure 3

Valence ratings associated with AU and BAP combinations. Note that the four facial-expression categories (negative, neutral, mildly positive, and highly positive) were rated in the intended order.

Figure 4

Protocol design, including setting the baseline of the fNIRS system, as well as windowing the data collection during the View and NF epochs to account for the delay in hemodynamic response.

Figure 5

Example of a successful block, where asymmetry during NF is significantly larger than during View.

Figure 6

The distribution of block success across subjects.

Figure 7

Distribution of the effect-size measure r in successful blocks.

Figure 8

Asymmetry change from View to NF in non-successful and successful blocks. Successful blocks are characterized with a marked asymmetry increase during NF, while non-successful ones are characterized with a slight decrease.

Figure 9

Left-lateralized increase in average oxygenated hemoglobin concentration (HbO) in successful blocks.

Figure 10

Mean and standard error of HbO across successful blocks (N = 70), for left (red) and right (blue) sides separately. The signal on the two sides overlaps completely during View, while left rises above right during NF.

Figure 11

Topographic snapshots of asymmetric HbO increase in the dorsolateral prefrontal cortex (DL-PFC) during a successful NF epoch.