Evolutionarily stable strategy analysis and its links to demography and genetics through invasion fitness

Abstract

Evolutionarily stable strategy (ESS) analysis pioneered by Maynard Smith and Price took off in part because it often does not require explicit assumptions about the genetics and demography of a population in contrast to population genetic models. Though this simplicity is useful, it obscures the degree to which ESS analysis applies to populations with more realistic genetics and demography: for example, how does ESS analysis handle complexities such as kin selection, group selection and variable environments when phenotypes are affected by multiple genes? In this paper, I review the history of the ESS concept and show how early uncertainty about the method lead to important mathematical theory linking ESS analysis to general population genetic models. I use this theory to emphasize the link between ESS analysis and the concept of invasion fitness. I give examples of how invasion fitness can measure kin selection, group selection and the evolution of linked modifier genes in response to variable environments. The ESSs in these examples depend crucially on demographic and genetic parameters, which highlights how ESS analysis will continue to be an important tool in understanding evolutionary patterns as new models address the increasing abundance of genetic and long-term demographic data in natural populations. This article is part of the theme issue 'Half a century of evolutionary games: a synthesis of theory, application and future directions'.

Keywords: evolutionarily stable strategy; group selection; kin selection; lineage fitness; reduction principle; variable environments.

Figure 1.

Pairwise invasibility plots for the mutation rate model of [168] when alternation between each of the two environments occurs every T = 50 generations. The fitness costs of being in the wrong environments are s₁ = 0.01 and s₂ = 0.015 and the recombination rate between the phenotypic locus and the modifier is given above each plot. White regions are where a mutant modifier allele with a mutation rate given on the vertical axis can invade a population fixed for a modifier allele with a mutation rate given on the horizontal axis, and black regions are where the mutant cannot invade; in other words, the leading eigenvalue of external stability matrix ${\mathcal{L}}_{\mathrm{ex}}$ in eqn. (17) of Liberman et al. [168] is greater than one in the white regions and less than one in the black regions.