An Exploratory Analysis of the Neural Correlates of Human-Robot Interactions With Functional Near Infrared Spectroscopy

Abstract

Functional near infrared spectroscopy (fNIRS) has been gaining increasing interest as a practical mobile functional brain imaging technology for understanding the neural correlates of social cognition and emotional processing in the human prefrontal cortex (PFC). Considering the cognitive complexity of human-robot interactions, the aim of this study was to explore the neural correlates of emotional processing of congruent and incongruent pairs of human and robot audio-visual stimuli in the human PFC with fNIRS methodology. Hemodynamic responses from the PFC region of 29 subjects were recorded with fNIRS during an experimental paradigm which consisted of auditory and visual presentation of human and robot stimuli. Distinct neural responses to human and robot stimuli were detected at the dorsolateral prefrontal cortex (DLPFC) and orbitofrontal cortex (OFC) regions. Presentation of robot voice elicited significantly less hemodynamic response than presentation of human voice in a left OFC channel. Meanwhile, processing of human faces elicited significantly higher hemodynamic activity when compared to processing of robot faces in two left DLPFC channels and a left OFC channel. Significant correlation between the hemodynamic and behavioral responses for the face-voice mismatch effect was found in the left OFC. Our results highlight the potential of fNIRS for unraveling the neural processing of human and robot audio-visual stimuli, which might enable optimization of social robot designs and contribute to elucidation of the neural processing of human and robot stimuli in the PFC in naturalistic conditions.

Keywords: face-voice matching; functional near infrared spectroscopy (fNIRS); hemodynamic; human-robot interaction; prefrontal cortex.

Figure 1

Human (A) and robot faces (B). Adapted from Mutlu et al. (2020).

Figure 2

Experimental design. (A) Experimental design of the fNIRS recording session, (B) Experimental design of the Behavioral session, (C) Time traces depicting the onsets and durations of each audio-visual stimulus during the fNIRS recording session. Four types of stimuli are repeated 10 times in a randomized order.

Figure 3

Time traces of block averaged Δ[HbO] signals of each audiovisual stimuli in representative channels, averaged across all subjects. Δ[HbO] resembles the relative concentration change of HbO in micromoles at probed brain tissue. Each time trace indicates block average of HbO responses detected during each specific audio-visual stimulus from the same channel of all participants. Pink shaded area represents the audio-visual stimulus interval. Error bars indicate the standard error of the mean signal across participants.

Figure 4

Changes in average Uncanniness score with respect to voice type and face type.

Figure 5

Channel locations of fNIRS probe mapped onto a standard brain template in MNI space. Channels with significant hemodynamic activation are illustrated with colored circles. (Red circle: Human Face > Robot Face, Green circle: Robot Voice < Human Voice, Blue circle: Human Face Robot Voice > Robot Face Robot Voice). Adapted from Mutlu et al. (2020).

Figure 6

Hemodynamic activation maps obtained during different contrasts between audiovisual stimuli. (A) Human Face > Robot Face contrast, (B) Robot Voice < Human Voice contrast, (C) Face-voice interaction: Human Face-Robot Voice > Robot Face-Robot Voice contrast. T scores of channels showing statistically significant activation (p < 0.05) are mapped on to the standard head model map. Color bar represents threshold t statistic scores.

Figure 7

Scatter plots of behavioral data (i.e., Uncanniness Rating) and Cohen's D metrics computed for Human Face - Robot Voice > Robot Face - Robot Voice contrast. Each dot represents a comparison of changes in behavioral score and hemodynamic parameter of channel 22 for each subject. The two variables were found to be linearly correlated (p < 0.05).