Testing for ancient admixture between closely related populations

Abstract

One enduring question in evolutionary biology is the extent of archaic admixture in the genomes of present-day populations. In this paper, we present a test for ancient admixture that exploits the asymmetry in the frequencies of the two nonconcordant gene trees in a three-population tree. This test was first applied to detect interbreeding between Neandertals and modern humans. We derive the analytic expectation of a test statistic, called the D statistic, which is sensitive to asymmetry under alternative demographic scenarios. We show that the D statistic is insensitive to some demographic assumptions such as ancestral population sizes and requires only the assumption that the ancestral populations were randomly mating. An important aspect of D statistics is that they can be used to detect archaic admixture even when no archaic sample is available. We explore the effect of sequencing error on the false-positive rate of the test for admixture, and we show how to estimate the proportion of archaic ancestry in the genomes of present-day populations. We also investigate a model of subdivision in ancestral populations that can result in D statistics that indicate recent admixture.

FIG. 1.

There are three possible topologies of the gene genealogies, I, II, and III, that can relate samples P₁, P₂, and P₃, and the outgroup O. Assuming that any polymorphisms reflect a single historical mutation, ABBA sites can occur only on a type III gene genealogy due to a mutation that occurs on the branch ancestral to samples P₂ and P₃ (IIIc), and BABA sites can occur only on a type II gene genealogy due to a mutation on the branch ancestral to samples P₁ and P₃ (IIc).

FIG. 2.

Model of IUA. P₁ and P₂ split at time t_P2. P₃ splits from the ancestral population of P₁ and P₂, denoted P₍₁₂₎, at time t_P3. A single episode of admixture takes place from P₃ to P₂ at time t_GF. We denote by f the admixture proportion, which is the proportion of P₃ ancestry in P₂ individuals at time t_GF. We denote by N₃(t), N₍₁₂₎(t), and N₍₁₂₃₎(t) the size of populations P₃, P₍₁₂₎, and P₍₁₂₃₎ at generation t in the past. The sizes of populations P₁ and P₂ do not influence D statistics.

FIG. 3.

Model of ancient subdivision (AS). P₁ and P₂ definitively split at time t_P2. The ancestral population of P₁ and P₂ is subdivided in two subpopulations, denoted P_(12),1 and P_(12),2. The two subpopulations exchange migrants at rate m. P₃ splits at time t_P3. The subdivision persists in the ancestral population of P_(12),1, P_(12),2, and P₃. We denote by P_(123),1 the population ancestral to P_(12),1 and by P_(123),2 the ancestral population of P_(12),2. These two subpopulations exchange migrants at rate m. Subpopulations merge at time T > t_P3. All population sizes are constant and equal to N.

FIG. 4.

D statistics and expected coalescence time of P₂ and P₃ under the IUA and AS models. We assumed parameter values of t_GF = 2,500, t_P2 = 3,000, t_P3 = 12,000, T = 25,000 (times in generations), and a constant ancestral population size equal to N = 10,000. (A) D(P₁, P₂, P₃, O) as a function of f (IUA) and 2Nm (AS). (B) Expected internal branch length (IBL) for topologies where P₂ and P₃ coalesced first as a function of f (IUA) and 2Nm (AS). The rescaling for 2Nm on (A) and (B) was chosen so that 2Nm and f vary in the same range.

FIG. 5.

D(P₁, P₂, P₃, O) as a function of the sequencing error difference e₁ − e₂. The dotted curve assumes that the probability to have an error in two populations is equal to the product of sequencing errors in the two populations (e_ij = e_i × e_j). The dashed curve corresponds to e_ij = min(e_i, e_j)/10, and the solid line assumes e_ij = min(e_i, e_j). The horizontal dashed line corresponds to twice the standard error of D, assuming that sites are independent.

FIG. 6.

Power of the test for admixture. We simulated (A) 10,000 unlinked polymorphic sites and (B) 100,000 unlinked polymorphic sites. Dots: no bottleneck and no migration between P₁ and P₂. Circles: bottleneck in the ancestral population of P₁ and P₂. Grey triangles: bottleneck in P₃. Squares: ongoing migration between P₁ and P₂ at rate 2Nm = 0.5. All bottlenecks involved a 100-fold reduction in population size and lasted 1,000 generations. They started at 3,500 generations in the past.