Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes

Abstract

The increasing amount of genomic data currently available is expanding the horizons of population genetics inference. A wide range of methods have been published allowing to detect and date major changes in population size during the history of species. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those generated under models of population size change. Recently, Mazet et al. (Heredity 116(4):362-371, 2016) introduced the idea that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size-change having an identical distribution of [Formula: see text] (the time of the first coalescence for a sample of size two). This implies that there is an identifiability problem between a panmictic and a structured model when we base our analysis only on [Formula: see text]. In this paper, based on an analytical study of the rate matrix of the ancestral lineage process, we obtain new theoretical results about the joint distribution of the coalescence times [Formula: see text] for a sample of three haploid genes in a n-island model with constant size. Even if, for any [Formula: see text], it is always possible to find a size-change scenario for a panmictic population such that the marginal distribution of [Formula: see text] is exactly the same as in a n-island model with constant population size, we show that the joint distribution of the coalescence times [Formula: see text] for a sample of three genes contains enough information to distinguish between a panmictic population and a n-island model of constant size.

Keywords: Demographic history; Inverse instantaneous coalescence rate (IICR); Population size change; Population structure; Rate matrix; Structured coalescent.