Coalescence times and FST under a skewed offspring distribution among individuals in a population

Abstract

Estimates of gene flow between subpopulations based on F(ST) (or N(ST)) are shown to be confounded by the reproduction parameters of a model of skewed offspring distribution. Genetic evidence of population subdivision can be observed even when gene flow is very high, if the offspring distribution is skewed. A skewed offspring distribution arises when individuals can have very many offspring with some probability. This leads to high probability of identity by descent within subpopulations and results in genetic heterogeneity between subpopulations even when Nm is very large. Thus, we consider a limiting model in which the rates of coalescence and migration can be much higher than for a Wright-Fisher population. We derive the densities of pairwise coalescence times and expressions for F(ST) and other statistics under both the finite island model and a many-demes limit model. The results can explain the observed genetic heterogeneity among subpopulations of certain marine organisms despite substantial gene flow.

Figure 1.—

The densities and of times to coalescence for two genes sampled from the same (T₀), or different (T₁), subpopulations as functions of time for different values of ψ when the number of subpopulations D = 3 and φ = κ = 1. The coalescence timescale is N²/2 in a and c and N^γ/2 with 0 < γ < 2 in b and d. The solid lines in a and c are the densities obtained under the standard coalescent (i.e., γ > 2).

Figure 2.—

The estimate of migration rate from Equation 11 as a function of ψ. (a) when F_ST = 0.1 (solid line), F_ST = 0.2 (dashed line), and F_ST = 0.5 (dotted line). (b) with F_ST = 0.1 and φ = 1 (solid line), φ = 2 (dashed line), and φ = 5 (dotted line).

Figure 3.—

The quantity F_ST from Equation 14 as a function of ψ (with φ = 1) for different values of θ, κ, and rate of coalescence (λ_γ). Solid lines, θ = 10; dashed lines, θ = 1; dotted lines, θ = 0.1.