Reinforcement Learning Approaches in Social Robotics

Abstract

This article surveys reinforcement learning approaches in social robotics. Reinforcement learning is a framework for decision-making problems in which an agent interacts through trial-and-error with its environment to discover an optimal behavior. Since interaction is a key component in both reinforcement learning and social robotics, it can be a well-suited approach for real-world interactions with physically embodied social robots. The scope of the paper is focused particularly on studies that include social physical robots and real-world human-robot interactions with users. We present a thorough analysis of reinforcement learning approaches in social robotics. In addition to a survey, we categorize existent reinforcement learning approaches based on the used method and the design of the reward mechanisms. Moreover, since communication capability is a prominent feature of social robots, we discuss and group the papers based on the communication medium used for reward formulation. Considering the importance of designing the reward function, we also provide a categorization of the papers based on the nature of the reward. This categorization includes three major themes: interactive reinforcement learning, intrinsically motivated methods, and task performance-driven methods. The benefits and challenges of reinforcement learning in social robotics, evaluation methods of the papers regarding whether or not they use subjective and algorithmic measures, a discussion in the view of real-world reinforcement learning challenges and proposed solutions, the points that remain to be explored, including the approaches that have thus far received less attention is also given in the paper. Thus, this paper aims to become a starting point for researchers interested in using and applying reinforcement learning methods in this particular research field.

Keywords: human-robot interaction; physical embodiment; reinforcement learning; reward design; social robotics.

Figure 1

Some of the social robots platforms referenced within the reviewed papers. (The pictures of (a) Pepper robot, and (b) Nao robot were taken by the authors. (c) Mini robot, the figure is adapted from [13]—licensed under the Creative Commons Attribution, (d) Maggie robot, the figure is from https://robots.ros.org/maggie/, accessed on 20 March 2020—licensed under the Creative Commons Attribution, (e) iCat robot, the figure is from https://www.bartneck.de/wp-content/uploads/2009/08/iCat02.jpg, accessed on 22 March 2020—used with permission, photo credit to Christoph Bartneck.)

Figure 2

A standard reinforcement learning framework (reproduced from [14] (p. 38)).

Figure 3

Taxonomy of Reinforcement Learning algorithms (reproduced and shortened from [37]).

Figure 4

Reinforcement Learning approaches in social robotics.

Figure 5

Interaction in Interactive Reinforcement Learning (reproduced from [96]).