Subdivision in an ancestral species creates asymmetry in gene trees

Abstract

We consider gene trees in three species for which the species tree is known. We show that population subdivision in ancestral species can lead to asymmetry in the frequencies of the two gene trees not concordant with the species tree and, if subdivision is extreme, cause the one of the nonconcordant gene trees to be more probable than the concordant gene tree. Although published data for the human-chimp-gorilla clade and for three species of Drosophila show asymmetry consistent with our model, sequencing error could also account for observed patterns. We show that substantial levels of persistent ancestral subdivision are needed to account for the observed levels of asymmetry found in these two studies.

FIG. 1.—

Illustration of the history of speciation and population subdivision for the analytic model developed in the text. Three species, A, B, and C are sampled today. The dashed line indicates a partial barrier to dispersal in the ancestral species. We define the parameter τ to be equal to the ratio of the internal branch (t₂ −t₃) to the time to the MRCA (t₂), τ =(t₂ − t₃)/t₂.

FIG. 2.—

Hypothetical series of events in a geographic region that could lead to the species tree shown in figure 1. The dashed line in the figure represents a partial barrier to dispersal and corresponds to the dashed line in figure 1. The solid lines represent vicariant events that isolate species on either side.

FIG. 3.—

One-locus probabilities of each of the three possible gene trees under the model of ancient subdivision across a range of migration rates. We assume the same population size and migration rate in S₄ and S₅. These results were obtained by numerical evaluation of equation (4) in the text.

FIG. 4.—

Two-locus simulations using 500,000 replicates were generated using Hudson's (2002) program ms under the demographic model described in figure 1 across a range of plausible recombination rates. The value of τ = 0.2 was used because it is roughly the value for the human–chimp–gorilla clade. Migration was enforced at the speciation event leading to species A and B and persisted at a constant rate back into the common ancestor of all three species. The function f is defined by equation (5) in the text.