Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study

Abstract

This paper presents our recent research on integrating artificial emotional intelligence in a social robot (Ryan) and studies the robot's effectiveness in engaging older adults. Ryan is a socially assistive robot designed to provide companionship for older adults with depression and dementia through conversation. We used two versions of Ryan for our study, empathic and non-empathic. The empathic Ryan utilizes a multimodal emotion recognition algorithm and a multimodal emotion expression system. Using different input modalities for emotion, i.e. facial expression and speech sentiment, the empathic Ryan detects users emotional state and utilizes an affective dialogue manager to generate a response. On the other hand, the non-empathic Ryan lacks facial expression and uses scripted dialogues that do not factor in the users emotional state. We studied these two versions of Ryan with 10 older adults living in a senior care facility. The statistically significant improvement in the users' reported face-scale mood measurement indicates an overall positive effect from the interaction with both the empathic and non-empathic versions of Ryan. However, the number of spoken words measurement and the exit survey analysis suggest that the users perceive the empathic Ryan as more engaging and likable.

Keywords: Artificial Emotional Intelligence; Dementia and Depression; Emotion Recognition; Social Robotics; empathic robots.

Fig. 1.

Using a multimodal emotion perception system to interpret the input modalities and output appropriate responses in a multimodal emotion expression system (SA: Sentiment Analysis, FER: Facial Expression Recognition).

Fig. 2.

Ryan’s animated face is capable of showing facial expressions.

Fig. 3.

The ResNet structure used for FER. The first few layers extracts the facial features and the Fully Connected Layers and the Softmax layer, classify the emotion. Layers in order from left to right: Input Image (64 × 64 × 3); Conv2D (64 × 64 × 16); 9 Residual Blocks (64 × 64 × 16); 1 Residual Block (32 × 32 × 32); 8 Residual Blocks (32 × 32 × 32); 1 Residual Block (16 × 16 × 64); 8 Residual Blocks (16 × 16 × 64); Fully Connected (3 outputs); Softmax (3 outputs).

Fig. 4.

The loss of the initial training phase (left) and fine tuning the network on images of people 50+ years old (right).

Fig. 5.

The emotion tracking system is more robust to sudden changes and noises in the input. The horizontal axis is time and the vertical axis is the emotional state with a range between −1 (Negative) and +1 (Positive).

Fig. 6.

Sample written dialogue between Ryan (blue) and a user (green). The sentiment of the user’s response is used to choose an empathic reply.

Fig. 7.

The architecture of the Ryan software. The module on the right is responsible for the dialogue. The modules on the left are responsible for sensing and expressing emotions.

Fig. 8.

Users interacting with Ryan.

Fig. 9.

Changes (improvement) in participants’ face-scale score after conversation with Ryan.