A Critical Period for Robust Curriculum-Based Deep Reinforcement Learning of Sequential Action in a Robot Arm

Abstract

Many everyday activities are sequential in nature. That is, they can be seen as a sequence of subactions and sometimes subgoals. In the motor execution of sequential action, context effects are observed in which later subactions modulate the execution of earlier subactions (e.g., reaching for an overturned mug, people will optimize their grasp to achieve a comfortable end state). A trajectory (movement) adaptation of an often-used paradigm in the study of sequential action, the serial response time task, showed several context effects of which centering behavior is of special interest. Centering behavior refers to the tendency (or strategy) of subjects to move their arm or mouse cursor to a position equidistant to all stimuli in the absence of predictive information, thereby reducing movement time to all possible targets. In the current study, we investigated sequential action in a virtual robotic agent trained using proximal policy optimization, a state-of-the-art deep reinforcement learning algorithm. The agent was trained to reach for appearing targets, similar to a serial response time task given to humans. We found that agents were more likely to develop centering behavior similar to human subjects after curricularized learning. In our curriculum, we first rewarded agents for reaching targets before introducing a penalty for energy expenditure. When the penalty was applied with no curriculum, many agents failed to learn the task due to a lack of action space exploration, resulting in high variability of agents' performance. Our findings suggest that in virtual agents, similar to infants, early energetic exploration can promote robust later learning. This may have the same effect as infants' curiosity-based learning by which they shape their own curriculum. However, introducing new goals cannot wait too long, as there may be critical periods in development after which agents (as humans) cannot flexibly learn to incorporate new objectives. These lessons are making their way into machine learning and offer exciting new avenues for studying both human and machine learning of sequential action.

Keywords: Curriculum learning; Movement optimization; Reinforcement learning; Robotic arm control; Sequential action.

Fig. 1

Participants spent increasingly more time during the ISI near the optimal position, equidistant from all possible stimulus locations. Distances in pixels. Shaded regions represent 95% CI. Data from de Kleijn et al. (2018b).

Fig. 2

The virtual environment with the target in green. The arm with a blue hand is moving toward the target. Targets could appear in one of four randomly chosen locations.

Fig. 3

Mean cumulative reward per episode. Shaded areas represent the SD.

Fig. 4

(a) Cumulative reward versus mean distance to optimal position and (b) cumulative reward versus total distance moved for 20 episodes from 5M to 6M for each run.

Fig. 5

Mean response time (in time steps) to touch the target. Shaded area represents SD. Note the log scale of the y‐axis.

Fig. 6

Absolute distance moved per time step (a) and absolute distance from the optimal position (b), minimal and equidistant from all possible target locations. Shaded regions represent SD.