Estimating FST and kinship for arbitrary population structures

Abstract

FST and kinship are key parameters often estimated in modern population genetics studies in order to quantitatively characterize structure and relatedness. Kinship matrices have also become a fundamental quantity used in genome-wide association studies and heritability estimation. The most frequently-used estimators of FST and kinship are method-of-moments estimators whose accuracies depend strongly on the existence of simple underlying forms of structure, such as the independent subpopulations model of non-overlapping, independently evolving subpopulations. However, modern data sets have revealed that these simple models of structure likely do not hold in many populations, including humans. In this work, we analyze the behavior of these estimators in the presence of arbitrarily-complex population structures, which results in an improved estimation framework specifically designed for arbitrary population structures. After generalizing the definition of FST to arbitrary population structures and establishing a framework for assessing bias and consistency of genome-wide estimators, we calculate the accuracy of existing FST and kinship estimators under arbitrary population structures, characterizing biases and estimation challenges unobserved under their originally-assumed models of structure. We then present our new approach, which consistently estimates kinship and FST when the minimum kinship value in the dataset is estimated consistently. We illustrate our results using simulated genotypes from an admixture model, constructing a one-dimensional geographic scenario that departs nontrivially from the independent subpopulations model. Our simulations reveal the potential for severe biases in estimates of existing approaches that are overcome by our new framework. This work may significantly improve future analyses that rely on accurate kinship and FST estimates.

Fig 1. Accuracy of F_ST and kinship estimators: Overview of models and results.

Our analysis is based on the generalized F_ST definition (section The generalized F_ST for arbitrary population structures) and two parallel models: the “Coancestry Model” for individual-specific allele frequencies (π_ij), and the “Kinship Model” for genotypes (x_ij). The “Coancestry in Terms of Kinship” panel connects kinship (${\phi }_{jk}^T$, $f_j^T$) and coancestry (${\theta }_{jk}^T$) parameters (section The kinship and coancestry models). We use these models to study the accuracy of F_ST and kinship method-of-moment estimators under arbitrary population structures. The “Indep. Subpop. F_ST Estimator” panel shows the bias resulting from the misapplication of F_ST estimators for independent subpopulations (${\widehat{F}}_{\text{ST}}^{\text{indep}}$) to arbitrary structures (section F_ST estimation based on the independent subpopulations model), as calculated under the coancestry model. The “Existing Kinship Estimator” panel shows the bias in the standard kinship model estimator (${\widehat{\phi }}_{jk}^{T,\text{std}}$) and its resulting plug-in F_ST estimator (${\widehat{F}}_{\text{ST}}^{\text{std}}$; section Characterizing a kinship estimator and its relationship to F_ST), as calculated under the kinship model. The “New Kinship Estimator” panel presents a new statistic A_jk that estimates kinship with a uniform bias, which together with a consistent estimator of its minimum value (${\widehat{A}}_{\text{min}}$) results in our new kinship (${\widehat{\phi }}_{jk}^{T,\text{new}}$) and F_ST (${\widehat{F}}_{\text{ST}}^{\text{new}}$) estimators, which are consistent under arbitrary population structure (section A new approach for kinship and F_ST estimation).

Fig 2. Coancestry matrices of simulations.

Both panels have n = 1000 individuals along both axes, K = 10 subpopulations (final or intermediate), and F_ST = 0.1. Color corresponds to ${\theta }_{jk}^T$ between individuals j and k (equal to ${\phi }_{jk}^T$ off-diagonal, $f_j^T$ along the diagonal). (A) The independent subpopulations model has ${\theta }_{jk}^T=0$ between subpopulations, and varying ${\theta }_{jj}^T$ per subpopulation, resulting in a block-diagonal coancestry matrix. (B) Our admixture scenario models a 1D geography with extensive admixture and intermediate subpopulation differentiation that increases with distance, resulting in a smooth coancestry matrix with no independent subpopulations (no ${\theta }_{jk}^T=0$ between blocks). Individuals are ordered along each axis by geographical position.

Fig 3. 1D admixture scenario.

We model a 1D geography population that departs strongly from the independent subpopulations model. (A) K = 10 intermediate subpopulations, evenly spaced on a line, evolved independently in the past with F_ST increasing with distance, which models a sequence of increasing founder effects (from left to right) to mimic the global human population. (B) Once differentiated, individuals in these intermediate subpopulations spread by random walk modeled by Normal densities. (C) n = 1000 individuals, sampled evenly in the same geographical range, are admixed proportionally to the previous Normal densities. Thus, each individual draws most of its alleles from the closest intermediate subpopulation, and draws the fewest alleles from the most distant populations. Long-distance random walks of intermediate subpopulation individuals results in kinship for admixed individuals that decays smoothly with distance in Fig 2B. (D) For F_ST estimators that require a partition of individuals into subpopulations, individuals are clustered by geographical position (K = 10).

Fig 4. Evaluation of F_ST estimators.

The Weir-Cockerham, Weir-Hill, Weir-Goudet (for individuals), HudsonK (equal to Weir-Goudet for subpopulations, S1 Text), BayeScan, ${\widehat{F}}_{\text{ST}}^{\text{std}}$ in Eq (25) derived from the standard kinship estimator, and our new F_ST estimator in Eqs (34) and (37), are evaluated on simulated genotypes from our two models (Fig 2). The Weir-Cockerham F_IT estimator was also included to show that estimation of total inbreeding behaves similarly to F_ST estimators. (A) The independent subpopulations model required by the Weir-Hill, HudsonK, and BayeScan F_ST estimators. All but standard kinship (${\widehat{F}}_{\text{ST}}^{\text{std}}$) and Weir-Goudet (for individuals) recover the target F_ST IBD probability in Eq (9) (red line) with small errors. (B) Our admixture scenario, which has no independent subpopulations, was constructed so ${\widehat{F}}_{\text{ST}}^{\text{std}}\approx \frac{1}{2}{F}_{\text{ST}}$. Only our new estimates are accurate. The rest of these estimators give values smaller than the target F_ST IBD probability, which result from treating kinship as zero between every subpopulations imposed by geographic clustering (or between individuals for Standard Kinship and Weir-Goudet). The ${\widehat{F}}_{\text{ST}}^{\text{indep}}$ estimator limit in Eq (14) (green dotted line) overlaps the true F_ST (red line) in (A) but not (B). Estimates (blue) include 95% prediction intervals (often too narrow to see) from 39 independently-simulated genotype matrices for each model (Methods, section Prediction intervals).

Fig 5. Evaluation of kinship estimators.

Observed accuracy for two existing kinship coefficient estimators is illustrated in our admixture simulation and contrasted to the nearly unbiased estimates of our new estimator. Plots show n = 1000 individuals along both axes, and color corresponds to ${\phi }_{jk}^T$ between individuals j ≠ k and to $f_j^T$ along the diagonal ($f_j^T$ is in the same scale as ${\phi }_{jk}^T$ for j ≠ k; plotting ${\phi }_{jj}^T$, which have a minimum value of $\frac{1}{2}$, would result in a discontinuity in this figure). (A) True kinship matrix. (B) Estimated kinship using our new estimator in Eqs (34) and (37) from simulated genotypes recovers the true kinship matrix with high accuracy. (C) Theoretical limit of ${\widehat{\phi }}_{jk}^{T,\text{std}}$ in Eq (19) as the number of independent loci goes to infinity demonstrates the accuracy of our bias predictions under the kinship model. (D) Standard kinship estimates ${\widehat{\phi }}_{jk}^{T,\text{std}}$ given by Eq (18) from simulated genotypes are downwardly biased on average and distorted by pair-specific amounts. (E) Theoretical limit of the Weir-Goudet kinship estimator given by Eq (38). (F) Weir-Goudet kinship estimates from the same simulated genotypes agree with our calculated limit.

Fig 6. Accuracy of kinship estimators.

Here the estimated kinship values are directly compared to their true values, in the same admixture simulation data (n = 1000 individuals) shown in the previous figure. (A) Kinship between different individuals (excluding inbreeding). The new estimator has practically no bias in this evaluation (falls on the 1-1 dashed gray line). The standard estimator has a complex, non-linear bias that covers a large area of errors. (B) Inbreeding comparison, shows the bias of the standard estimate follows a different pattern for inbreeding compared to kinship between individuals. To better visualize and compare data across panels, a random subset of n points (out of the original n(n − 1)/2 unique individual pairs) were plotted in (A), matching the number of individuals (number of points in (B)).

Fig 7. Evaluation of standard and adjusted F_ST estimators.

The convergence values we calculated for the standard kinship plug-in and adjusted F_ST estimators are validated using our admixture simulation. All adjusted estimators are unbiased but are “oracle” methods, since the mean kinship (${\overline{\phi }}^T$), mean coancestry (${\overline{\theta }}^T$), or bias coefficient ($s^T=\frac{{\overline{\theta }}^T}{F_{\text{ST}}}$ for IAFs, replaced by $\frac{{\overline{\phi }}^T}{F_{\text{ST}}}$ for genotypes) are usually unknown. (A) Estimation from individual-specific allele frequencies (IAFs): ${\widehat{F}}_{\text{ST}}^{\text{std}}$ is the standard coancestry plug-in estimator in Eq (26); ${\widehat{F}}_{\text{ST}}^{\prime }$ “Adj. ${\overline{\theta }}^T$” is in Eq (27); ${\widehat{F}}_{\text{ST}}^{\prime \prime }$ “Adj. s” is in Eq (31). (B) For genotypes, ${\widehat{F}}_{\text{ST}}^{\text{std}}$ is given in Eq (25), and the adjusted estimators use ${\overline{\phi }}^T$ rather than ${\overline{\theta }}^T$. Lines: true F_ST (red line), limits of biased estimators ${\widehat{F}}_{\text{ST}}^{\text{std}}$ (green lines, which differ slightly per panel). Estimates (blue) include 95% prediction intervals (too narrow to see) from 39 independently-simulated genotype matrices for our admixture model (Methods, section Prediction intervals).