The effects of task difficulty, novelty and the size of the search space on intrinsically motivated exploration

Abstract

Devising efficient strategies for exploration in large open-ended spaces is one of the most difficult computational problems of intelligent organisms. Because the available rewards are ambiguous or unknown during the exploratory phase, subjects must act in intrinsically motivated fashion. However, a vast majority of behavioral and neural studies to date have focused on decision making in reward-based tasks, and the rules guiding intrinsically motivated exploration remain largely unknown. To examine this question we developed a paradigm for systematically testing the choices of human observers in a free play context. Adult subjects played a series of short computer games of variable difficulty, and freely choose which game they wished to sample without external guidance or physical rewards. Subjects performed the task in three distinct conditions where they sampled from a small or a large choice set (7 vs. 64 possible levels of difficulty), and where they did or did not have the possibility to sample new games at a constant level of difficulty. We show that despite the absence of external constraints, the subjects spontaneously adopted a structured exploration strategy whereby they (1) started with easier games and progressed to more difficult games, (2) sampled the entire choice set including extremely difficult games that could not be learnt, (3) repeated moderately and high difficulty games much more frequently than was predicted by chance, and (4) had higher repetition rates and chose higher speeds if they could generate new sequences at a constant level of difficulty. The results suggest that intrinsically motivated exploration is shaped by several factors including task difficulty, novelty and the size of the choice set, and these come into play to serve two internal goals-maximize the subjects' knowledge of the available tasks (exploring the limits of the task set), and maximize their competence (performance and skills) across the task set.

Keywords: decision making; exploration; intrinsic motivation; novelty; video games.

Figure 1

Task design. (A) Individual game. The subjects pressed a key to intercept a stream of moving dots (arrow) as they crossed the screen center. (B–D) Selection screens in the 7-game, 64-game, and 64N-game versions.

Figure 2

General performance. (A) Performance as a function of speed in the 3 versions. Each bin represents the average and standard error (s.e.m.) of the fraction correct for the corresponding dot speed across all the subjects tested. (B) The distribution of performance levels in the 7-game condition. The points show the average and s.e.m. (across subjects) of the number of games in each of 6 performance bins.

Figure 3

Selection and performance in the 3 task versions. (A) Evolution of the selected speed during a session. Each colormap indicates the probability of selection of a given speed, measured across all subjects in a sliding window over the session. The bottom panel shows the average dot speed in each time bin (average and s.e.m. from the corresponding colormaps). (B) Evolution of performance during a session. Same as in (A), except that the grayscale indicates the probability of playing at a given fraction correct in each time bin.

Figure 4

Range of selected games. (A) Distribution of the selected speeds (top) and fraction correct (bottom) across an entire session. The values show the mean and s.e.m. across subjects. (B) Choices of individual subjects. Each line represents one subject and shows the maximum, minimum and average dot speed selected by that subject. Subjects are ordered according to the task version (or task combination) that they performed, and in chronological order within a task group.

Figure 5

Local strategy for game selection. Each point shows the average and s.e.m. of the probability to repeat, increase or decrease difficulty as a function of prior game performance. Solid colored traces show the empirical data, dotted black traces show the results of simulations using a random game selection strategy.

Figure 6

Repetition rates. (A) The likelihood of repeating a speed calculated as in Figure 5, but only for the first 1/3 of games in each session. (B) The fraction of repetitions, in the 64N versions, where subjects requested the same or a novel sequence. The points show the mean and s.e.m., across subjects, of the tendency to choose a new or familiar sequence when repeating a game of given speed.

Figure 7

Subjective rating of learning progress in the 64-game task. (A) Subjective improvement rating as a function of actual improvement. Each point represents a game that was rated by a subject after the end of the session, and the data are pooled across subjects. The y axis shows the subject's rating of his/her own improvement and the x axis shows the objective improvement in units of %correct/game. The lines show best fit linear regression across subjects. (B) The probability of selecting a game as a function of the subjective improvement rating. (C) Subjective rating as a function of average performance.