Improving Robot Motor Learning with Negatively Valenced Reinforcement Signals

Abstract

Both nociception and punishment signals have been used in robotics. However, the potential for using these negatively valenced types of reinforcement learning signals for robot learning has not been exploited in detail yet. Nociceptive signals are primarily used as triggers of preprogrammed action sequences. Punishment signals are typically disembodied, i.e., with no or little relation to the agent-intrinsic limitations, and they are often used to impose behavioral constraints. Here, we provide an alternative approach for nociceptive signals as drivers of learning rather than simple triggers of preprogrammed behavior. Explicitly, we use nociception to expand the state space while we use punishment as a negative reinforcement learning signal. We compare the performance-in terms of task error, the amount of perceived nociception, and length of learned action sequences-of different neural networks imbued with punishment-based reinforcement signals for inverse kinematic learning. We contrast the performance of a version of the neural network that receives nociceptive inputs to that without such a process. Furthermore, we provide evidence that nociception can improve learning-making the algorithm more robust against network initializations-as well as behavioral performance by reducing the task error, perceived nociception, and length of learned action sequences. Moreover, we provide evidence that punishment, at least as typically used within reinforcement learning applications, may be detrimental in all relevant metrics.

Keywords: inverse kinematics; nociception; punishment; reinforcement learning; self-protective mechanisms.

Figure 1

(A) Top view of the NAO robot facing left. The left arm is depicted in different positions and a blue line is superimposed to indicate the boundaries of the end-effector workspace. (B) Depiction of the target and end-effector coordinates of the randomly generated training set. Blue dots represent targets, whereas red asterisks represent end-effector initial positions. (C) The histograms show the initial distance between the end-effector and the corresponding target.

Figure 2

The neural architecture used for inverse kinematics learning. For clarity, only one connection weight is shown (arrow between neuron layers). The hidden layers for both the Actor (left-hand side) and the Critic (right-hand side) are independently tuned. Solid units and connection weights in black correspond to the baseline, i.e., the reward only condition, and are extended by the other 3 conditions. The punishment feedback given to the critic and depicted in red is only used for the reward + punishment, and reward + punishment + nociception conditions. Blue dashed units and blue dashed connection weights are only considered under the reward + nociception, and reward + punishment + nociception conditions. During training a* is performed. a* is determined based on the exploration of action a as described in equation (2). The Critic is trained every time step based on the TD error δ, while the actor is trained only if the TD error is positive.

Figure 3

A representative example of all four conditions of the fitness distribution over generations; the results for each particular condition can be found in the Supplementary Material Navarro-Guerrero et al., . The fitness is directly computed from the total distance to the target on the testing set once learning has been concluded, thus the lower the value the better.

Figure 4

Mean positioning error of the best individuals of each condition. All conditions but reward + punishment converge fast and reach small positioning error in a small number of epochs.

Figure 5

Average measured nociception for both joints, for all samples in the validation set and for the best individuals of each condition. All four conditions show an oscillatory convergent behavior. The condition trained with reward and nociceptive units converges to a smaller value than all other conditions.

Figure 6

Average number of steps needed to position the robot’s arm for the best individuals of each condition.