The Difference in the Proportions of Deleterious Variations within and between Populations Influences the Estimation of FST

Abstract

Estimating the extent of genetic differentiation between populations is an important measure in population genetics, ecology and evolutionary biology. The fixation index, or FST, is an important measure, which is routinely used to quantify this. Previous studies have shown that the FST estimated for selectively constrained regions was significantly lower than that estimated for neutral regions. By deriving the theoretical relationship between FST at neutral and constrained sites, we show that excess in the fraction of deleterious variations segregating within populations compared to those segregating between populations is the cause for the reduction in FST estimated at constrained sites. Using whole-genome data, our results revealed that the magnitude of reduction in FST estimates obtained for selectively constrained regions was much higher for distantly related populations compared to those estimated for closely related pairs. For example, the reduction was 47% for comparison between Europeans and Africans, 30% for the European and Asian comparison, 16% for the Northern and Southern European pair, and only 4% for the comparison involving two Southern European (Italian and Spanish) populations. Since deleterious variants are purged over time due to purifying selection, their contribution to the among-population diversity at constrained sites decreases with the increase in the divergence between populations. However, within-population diversities remain the same for all pairs compared; therefore, the FST estimated at constrained sites for distantly related populations are much smaller than those estimated for closely related populations. We obtained similar results when only the SNPs with similar allele frequencies at neutral and constrained sites were used. Our results suggest that the level of population divergence should be considered when comparing constrained site FST estimates from different pairs of populations.

Keywords: FST; deleterious mutations; population differentiation; temporal distributions and population genetics theory.

Figure 1

(A) Relationship between selection intensity and F_ST. Whole-exome data comprising synonymous SNPs (sSNPs) and nonsynonymous SNPs (nSNPs) for the Italian (TSI)–Nigerian (YRI) population pair was used to estimate F_ST. The magnitude of selection intensity on nSNPs was measured by the Combined Annotation-Dependent Depletion (CADD) method that integrates many diverse annotations into a single measure (C score) [31]. A bootstrap resampling procedure (1000 replicates) was used to estimate the standard error. (B) The magnitude of the reduction in the F_ST estimates and selection intensity. The X-axis shows the reduction in F_ST estimates of nSNPs in comparison with that of sSNPs (ρ) using Equation (11) (see Materials and Methods) for the exome data described above. Error bars show the standard error of the mean.

Figure 2

The theoretical relationship between the excess in the fraction of deleterious mutations segregating within population compared to between populations (η) and the magnitude of the reduction in the F_ST estimates of nSNPs (ρ) using Equation (12) (see Materials and Methods). The line was plotted using within- and between-population heterozygosities of neutral sSNPs for the Italian–Nigerian comparison and the red dots are the ρ values estimated from the exome data using Equation (11). Using the theoretical expected line, η values were predicted for the corresponding observed ρ values.

Figure 3

F_ST estimates for synonymous and highly constrained nonsynonymous SNPs of the (A) Italian–Nigerian, (B) Italian–Chinese, (C) Italian–British, and (D) Italian–Spanish population pairs. Error bars are the standard error of the mean, and a bootstrap resampling procedure (1000 replicates) was used to estimate the variance. The difference between the FST estimates of the neutral and constrained sites are highly significant (p < 0.01, Z test) for three comparisons and not significant for the Italian–Spanish pair.

Figure 4

The magnitude of the reduction in F_ST estimates of the nSNPs obtained for four population pairs. The population tree on top is drawn to highlight the correlation between the population divergence and the magnitude of the reduction in F_ST.

Figure 5

The theoretical relationship between the excess in the fraction of deleterious mutations segregating within population and those between populations (η) and the magnitude of the reduction in the F_ST estimates of nSNPs (ρ) using Equation (12) (see Materials and Methods). Neutral population diversities based on sSNPs for within- and between-population comparisons of the Italian–Nigerian, Italian–Chinese, Italian–British, and Italian–Spanish pairs were used to plot the lines, and the ρ estimated from the exome data (using Equation (11)) are shown as red closed circles. The theoretical lines were used to predict the η values of the corresponding ρ estimated using the exome data.