Design of a Virtual Reality System for Affect Analysis in Facial Expressions (VR-SAAFE); Application to Schizophrenia

Abstract

Schizophrenia is a life-long, debilitating psychotic disorder with poor outcome that affects about 1% of the population. Although pharmacotherapy can alleviate some of the acute psychotic symptoms, residual social impairments present a significant barrier that prevents successful rehabilitation. With limited resources and access to social skills training opportunities, innovative technology has emerged as a potentially powerful tool for intervention. In this paper, we present a novel virtual reality (VR)-based system for understanding facial emotion processing impairments that may lead to poor social outcome in schizophrenia. We henceforth call it a VR System for Affect Analysis in Facial Expressions (VR-SAAFE). This system integrates a VR-based task presentation platform that can minutely control facial expressions of an avatar with or without accompanying verbal interaction, with an eye-tracker to quantitatively measure a participants real-time gaze and a set of physiological sensors to infer his/her affective states to allow in-depth understanding of the emotion recognition mechanism of patients with schizophrenia based on quantitative metrics. A usability study with 12 patients with schizophrenia and 12 healthy controls was conducted to examine processing of the emotional faces. Preliminary results indicated that there were significant differences in the way patients with schizophrenia processed and responded towards the emotional faces presented in the VR environment compared with healthy control participants. The preliminary results underscore the utility of such a VR-based system that enables precise and quantitative assessment of social skill deficits in patients with schizophrenia.

Fig. 1.

Overall system diagram.

Fig. 2.

The eye tracker application. The application computed the three basic eye indices, i.e. pupil diameter (PD), blink rate (BR), and fixation duration (FD) online and logs them further offline analysis.

Fig. 3.

Peripheral physiological electrodes placement. The sensors included 3 EMG sensors, two on the face and one on the top shoulder as shown in the figure, PPG, GSR and SKT in the fingers with the two black strips showing the bipolar electrodes placed in the distal phalanx of the index and ring fingers with SKT at the middle finger and PPG in the thumb.

Fig. 4.

Masked maps overlaid on at the side of heat map visualizations. The top figure shows the SZ group gaze while the lower figure depicts the CTR group gaze aggregated across subjects and trials.

Fig. 5.

Physiological clustering using the density peaks method. Positive vs. negative categories for SZ (top) and CTR (bottom).

Fig. 6.

Clustering accuracies using the density peaks method.

Fig. 7.

Overall performance metrics.

Fig. 8.

Per emotion raw (biased) and bias-corrected performance.