The evolutionary consequences of learning under competition

Abstract

Learning is a taxonomically widespread process by which animals change their behavioural responses to stimuli as a result of experience. In this way, it plays a crucial role in the development of individual behaviour and underpins substantial phenotypic variation within populations. Nevertheless, the impact of learning in social contexts on evolutionary change is not well understood. Here, we develop game theoretical models of competition for resources in small groups (e.g. producer-scrounger and hawk-dove games) in which actions are controlled by reinforcement learning and show that biases in the subjective valuation of different actions readily evolve. Moreover, in many cases, the convergence stable levels of bias exist at fitness minima and therefore lead to disruptive selection on learning rules and, potentially, to the evolution of genetic polymorphisms. Thus, we show how reinforcement learning in social contexts can be a driver of evolutionary diversification. In addition, we consider the evolution of ability in our games, showing that learning can also drive disruptive selection on the ability to perform a task.

Keywords: disruptive selection; fitness minima; hawk–dove game; negative frequency dependence; producer–scrounger game; reinforcement learning.

Figure 1.

The pay-off rate to a mutant that has a fixed probability, $p$ , of taking action $u_2$ in the symmetric game. (i) Residents choose action $u_2$ with probability $0.5$ in each round (horizontal dashed line). (ii) Residents learn using an unbiased learning rule. Two learning rules are illustrated: AV learning with $T=400$ (red crosses) and AC learning with $T=100$ (blue open circles). $G=10$ .

Figure 2.

Adaptive dynamics for the symmetric game. (a) and (b) give results for AV learning with $T=400$ . (a) The strength of selection (value of the selection gradient) on the inflation bias $\alpha$ , showing that $\alpha =0$ is a convergence stable point. (b) The pay-off rate to a mutant when the resident strategy is $\alpha =0$ . (c) and (d) give results for AC learning with $T=100$ . (c) The strength of selection on the initial bias ${\theta }_0$ , showing that ${\theta }_0=0$ is a convergence stable point. (d) The pay-off rate to a mutant when the resident strategy is ${\theta }_0=0$ . $G=10$ .

Figure 3.

Adaptive dynamics for the producer–scrounger game. (a) and (b) give results for AV learning with $T=400$ . (a) The strength of selection on the inflation bias $\alpha$ , showing that there is a convergence stable point at approximately $\alpha =-0.05422$ . (b) The pay-off rate to a mutant when the resident strategy is $\alpha =-0.05422$ . (c) and (d) give results for AC learning with $T=100$ . (c) The strength of selection on the initial bias ${\theta }_0$ showing a convergence stable point at approximately ${\theta }_0=-2.688$ . (d) The pay-off rate to a mutant when the resident strategy is ${\theta }_0=-2.688$ . $G=10$ . Foraging parameters $e_p=2$ , $e_s=3$ .

Figure 4.

Distribution of the evolved bias in the producer–scrounger game after 50 000 generations. (a) Inflation bias $\alpha$ under AV learning when reproduction is sexual. (b) Initial bias ${\theta }_0$ under AC learning when reproduction is sexual. (c) Inflation bias $\alpha$ under AV learning when reproduction is asexual. (d) Initial bias ${\theta }_0$ under AC learning when reproduction is asexual. $T=400$ for AV learning $T=100$ for AC learning. $G=10$ . Foraging parameters $e_p=2$ , $e_s=3$ . Details of the evolutionary simulation are given in electronic supplementary material, §4.

Figure 5.

Ability bias in the producer–scrounger and hawk–dove games. (a) Strength of selection on ability bias $a$ in the producer–scrounger game, showing a convergence stable value at approximately $a^{*}=-0.2897$ . (b) The pay-off to a mutant with given ability bias when the resident population has ability bias $a=a^{*}$ in the producer–scrounger game. Two cases are illustrated: (i) there is no initial bias during learning ( ${\theta }_0=0$ , blue squares) and (ii) a mutant with ability bias $a$ has initial bias ${\theta }_0=-0.4(a-a^{*})$ (black triangles). (c) and (d) give analogous results for the hawk–dove game, for which $a^{*}=0.218$ . AC learning with $T=100$ . $e_p=2$ , $e_s=3$ , $V=2,C=4$ , $G=10$ .