An Improved F(st) Estimator

Abstract

The fixation index F(st) plays a central role in ecological and evolutionary genetic studies. The estimators of Wright ([Formula: see text]), Weir and Cockerham ([Formula: see text]), and Hudson et al. ([Formula: see text]) are widely used to measure genetic differences among different populations, but all have limitations. We propose a minimum variance estimator [Formula: see text] using [Formula: see text] and [Formula: see text]. We tested [Formula: see text] in simulations and applied it to 120 unrelated East African individuals from Ethiopia and 11 subpopulations in HapMap 3 with 464,642 SNPs. Our simulation study showed that [Formula: see text] has smaller bias than [Formula: see text] for small sample sizes and smaller bias than [Formula: see text] for large sample sizes. Also, [Formula: see text] has smaller variance than [Formula: see text] for small Fst values and smaller variance than [Formula: see text] for large F(st) values. We demonstrated that approximately 30 subpopulations and 30 individuals per subpopulation are required in order to accurately estimate F(st).

Fig 1. The relationship between F^st and ϑ^ for simulated data.

The x-axis shows the difference of allele frequencies between two subpopulations $\hat{\vartheta }$ (left plots) and ${\hat{\vartheta }}^2$ (right plots); the y-axis shows ${\hat{F}}_{st}$ values for Wright’s (top row), Weir and Cockerham’s (second row), the modified (third row), and Hudson et al.’s estimators (bottom row), and the legend indicates the sample sizes n ₁ (before hyphen) and n ₂ (after hyphen).

Fig 2. Bias as a function of total sample size.

The x-axis shows the total sample size (n ₁ + n ₂). The y-axis shows $F_{st}-{\hat{F}}_{s{t}_1}$ (red), $F_{st}-{\hat{F}}_{s{t}_2}$ (blue), $F_{st}-{\hat{F}}_{s{t}_m}$ (green), and $F_{st}-{\hat{F}}_{s{t}_3}$ (orange) for r = 2.

Fig 3. Effect of the number of subpopulations on bias.

The x-axis shows the number of subpopulations. The y-axis shows the mean (left) and variance (right) of $F_{st}-{\hat{F}}_{s{t}_1}$ (red), $F_{st}-{\hat{F}}_{s{t}_2}$ (blue), $F_{st}-{\hat{F}}_{s{t}_m}$ (green), and $F_{st}-{\hat{F}}_{s{t}_3}$ (orange) values, given F _st = 0.5 and average allele frequency p = 0.2. The top plot represents 5 individuals per subpopulation and the bottom plot represents 1000 individuals per subpopulation.

Fig 4. Effect of the number of individuals per subpopulation on bias.

The x-axis shows the number of individuals per subpopulation. The y-axis shows the mean (left) and variance (right) of $F_{st}-{\hat{F}}_{s{t}_1}$ (red), $F_{st}-{\hat{F}}_{s{t}_2}$ (blue), $F_{st}-{\hat{F}}_{s{t}_m}$ (green), and $F_{st}-{\hat{F}}_{s{t}_3}$ (orange) values, given F _st = 0.5 and an average allele frequency p = 0.2. From top to bottom, the plots represent the number of subpopulations r = 10, 20, and 40, respectively.