GST' , Jost's D, and FST are similarly constrained by allele frequencies: A mathematical, simulation, and empirical study

Abstract

Statistics $G_{\mathrm{ST}}^{\text{'}}$ and Jost's D have been proposed for replacing F_ST as measures of genetic differentiation. A principal argument in favour of these statistics is the independence of their maximal values with respect to the subpopulation heterozygosity H_S , a property not shared by F_ST . Nevertheless, it has been unclear if these alternative differentiation measures are constrained by other aspects of the allele frequencies. Here, for biallelic markers, we study the mathematical properties of the maximal values of $G_{\mathrm{ST}}^{\text{'}}$ and D, comparing them to those of F_ST . We show that $G_{\mathrm{ST}}^{\text{'}}$ and D exhibit the same peculiar frequency-dependence phenomena as F_ST , including a maximal value as a function of the frequency of the most frequent allele that lies well below one. Although the functions describing $G_{\mathrm{ST}}^{\text{'}}$ , D, and F_ST in terms of the frequency of the most frequent allele are different, the allele frequencies that maximize them are identical. Moreover, we show using coalescent simulations that when taking into account the specific maximal values of the three statistics, their behaviours become similar across a large range of migration rates. We use our results to explain two empirical patterns: the similar values of the three statistics among North American wolves, and the low D values compared to $G_{\mathrm{ST}}^{\text{'}}$ and F_ST in Atlantic salmon. The results suggest that the three statistics are often predictably similar, so that they can make quite similar contributions to data analysis. When they are not similar, the difference can be understood in relation to features of genetic diversity.

Keywords: allele frequency; gene flow; genetic differentiation; migration; population structure.

FIGURE 1

Range of possible values of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D as functions of the frequency M of the most frequent allele, for different numbers of subpopulations K. The shaded region represents the space between the minimal and maximal values. The maximal $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D are computed from Equations 9–11, respectively. The dashed line represents 1 for $F_{\text{ST}}$ and $G_{\text{ST}}^{\prime }$, and 2KM(1 − M)/(K − 1) for D (Equation S1.4 in Supporting Information File S1); the maximum value touches the dashed line when M = i/K for integers i in $\left[\lceil \frac{K}{2}\rceil ,K-1\right].$ For $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D, for each K, the minimum value is 0 for all values of M

FIGURE 2

The means A_F, A_G, and A_D of the maximal values of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D, respectively, over the interval M ∈ [1/2, 1), as functions of the number of subpopulations K. A_F(K) is computed from Equation 12, A_G(K) from Equation 13, and A_D(K) from Equation 14. The x-axis is plotted on a logarithmic scale

FIGURE 3

Joint density of the frequency M of the most frequent allele and statistics $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D, for different scaled migration rates 4Nm, considering K = 2 subpopulations. The black solid line represents the maximum value of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, or D in terms of M (Equations 9–11); the red dashed line represents the mean $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D in sliding windows of M of size 0.02 (plotted from 0.51 to 0.99). Colours represent the density of loci, estimated using a Gaussian kernel density estimate with a bandwidth of 0.007, with density set to 0 outside the minimum and maximum values. Loci are simulated using coalescent software MS, assuming an island model of migration and conditioning on one segregating site. Each panel considers 100,000 replicate simulations, with 100 lineages sampled per subpopulation

FIGURE 4

Joint density of the frequency M of the most frequent allele and statistics $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D, for different scaled migration rates 4Nm, considering K = 7 subpopulations. The simulation procedure and figure design follow Figure 3

FIGURE 5

Mean ${\overline{F}}_{\text{ST}},{\overline{G}}_{\text{ST}}^{\prime },$ and D across biallelic loci. (a) Unnormalized means ${\overline{F}}_{\text{ST}},{\overline{G}}_{\text{ST}}^{\prime },$ and D. (b) Normalized means ${\overline{F}}_{\text{ST}}/{\overline{F}}_{\max },{\overline{G}}_{\text{ST}}^{\prime }/{\overline{G}}_{\max }^{\prime }$ and $\overline{D}/{\overline{D}}_{\max },$ the ratio of the mean value to the mean maximal value given the observed frequency M of the most frequent allele. Both plots show quantities as functions of the number of subpopulations K and the scaled migration rate 4Nm. Colours represent the different statistics. Line types represent values of K: 2 (solid), 7 (dashed), and 40 (dotted). Values are computed from coalescent simulations using software ms as in Figure 3, with 1,000 replicate biallelic loci and 100 lineages per subpopulation. ${\overline{F}}_{\max },{\overline{G}}_{\max }^{\prime }$ and < ${\overline{D}}_{\max }$ are respectively computed from equation 11 of Alcala and Rosenberg (2017) and Equations D3 and D4 in Appendix D

FIGURE 6

Joint density of the frequency M of the most frequent allele and three differentiation measures ($F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D), and unnormalized and normalized mean values of the differentiation measures across loci, for 305 wolves and 91 dogs from North America, using 123,801 SNPs. (a) M and $F_{\text{ST}}$. (b) M and $G_{\text{ST}}^{\prime }.$ (c) M and D. (d) Unnormalized mean values of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D across SNPs, and the mean values of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D across SNPs normalized by the mean of their maximal values. In (a-c), the figure design follows Figure 3. In (d), ${\overline{F}}_{\max },{\overline{G}}_{\max }^{\prime },$ and ${\overline{D}}_{\max }$ are respectively computed from equation 11 of Alcala and Rosenberg (2017) and Equations D3 and D4 in Appendix D

FIGURE 7

Joint density of the frequency M of the most frequent allele and three differentiation measures ($F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D), and unnormalized and normalized mean values of the differentiation measures across loci, for 900 Atlantic salmon from 26 populations, using 1,335 SNPs. Sample sizes range from 25 to 40 per population. (a) M and $F_{\text{ST}}$. (b) M and $G_{\text{ST}}^{\prime }$. (c) M and D. (d) Mean values of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime },$ and D across SNPs, for sets of geographic regions as a function of K, the number of regions considered. (e) Ratio of mean values of $F_{\text{ST}}$, $G_{\text{ST}}^{\prime }$, and D across SNPs to their maximal mean values as functions of K. In (a-c), the figure design follows Figure 3. Coloured bars in (d) and (e) represent 2.5 and 97.5 quantiles of distributions of values across sets of size K. In (e), ${\overline{F}}_{\max },{\overline{G}}_{\max }^{\prime },\text{and}{\overline{D}}_{\max }$ are respectively computed from equation 11 of Alcala and Rosenberg (2017) and Equations D3 and D4 in Appendix D