'Highly-Informative' Genetic Markers Can Bias Conclusions: Examples and General Solutions

Abstract

High-grading bias is the overestimation power in a subset of loci caused by model overfitting. Using both empirical and simulated datasets, we show that high-grading bias can cause severe overestimation of population structure, and thus mislead investigators, whenever highly informative or high-F_ST markers are chosen (i.e., ascertained) and used for subsequent assessments, a common practice in population genetic studies. This problem can occur in panmictic populations with no local adaptation. Biased results from choosing high-F_ST markers may have severe downstream implications for management and conservation, such as erroneous conservation unit delineation, which could squander limited conservation resources to protect incorrectly defined 'populations'. Furthermore, we caution that high-grading is not limited to F_ST approaches; high-grading bias is a concern whenever a small subset of markers are first chosen to explain differences among groups based on their degree of difference and are subsequently reused to estimate the degree of difference among those groups. For example, selecting high F_ST loci for use in a GT-seq panel or using differentially expressed genes to plot sample membership in multivariate space can both result in spurious structure when none exists. We illustrate that using statistically based outlier tests in place of arbitrary F_ST cut-offs can reduce bias. Alternatively, permutation tests or cross-evaluation can be used to detect high-grading bias. We provide an R package, PCAssess, to help researchers detect and prevent high-grading bias in genetic datasets by automating permutation tests and principal component analyses (https://github.com/hemstrow/PCAssess).

Keywords: ecological genetics; genomics/proteomics; natural selection and contemporary evolution; population genetics—theoretical.

FIGURE 1

High‐grading bias in clustering algorithms and population assignment using empirical data from a single, panmictic population of monarch butterflies (i.e., no real population structure). We randomly assigned individuals to one of four artificially created populations and detected erroneous structure when the top 5% of SNPs with the highest F _ST values were used. (A–C) Population structure analysis results using all the SNPs found in this population (63,514 loci), and panels (D–F) illustrate population structure analyses using the highest‐F _ST dataset, where only the top 5% of SNPs with the highest F _ST were used. Panels (A) and (D) illustrate results using PCA, panels (B) and (E) illustrate results using STRUCTURE with k = 4 and 20,000 burn‐in and 100,000 MCMC iterations, and panels C and F illustrate results using the ‘self_assign’ function in the Bayesian assignment R package Rubias with the default parameters.

FIGURE 2

Conceptual figure illustrating a permutation approach that can be used to quantify high‐grading bias. In each permutation, the population IDs are randomly shuffled (without replacement), F _ST recalculated, high‐F _ST loci chosen, and F‐statistics calculated. The empirically observed change in clustering due to high‐grading (ΔF) is compared to a null distribution of ΔF from the 1000 permutations. A large ΔF represents a large observed increase in within‐group clustering when the highest‐F _ST loci are used, and a smaller or negative ΔF represents less of an increase or a decrease in clustering when the highest‐F _ST loci are used. Comparing ΔF values between empirical datasets and those where population IDs are permuted thousands of times allows for the detection of high‐grading bias.

FIGURE 3

Statistical detection of erroneous clustering due to high‐grading bias when high‐F _ST loci are used in PCAs using simulated data for three commonly observed types of population structure: Panmictic (without selection), high gene flow (without selection), and high gene flow with local adaptation (with selection). Panels A–C show PCAs constructed from all SNPs, and panels (D–F) show PCAs constructed from the top 5% of high‐F _ST SNPs. Note that all three scenarios show population structure in the latter case, even when none should exist (as in the panmictic scenario). Panels G–I depict the observed increase in clustering (ΔF) between the PCAs for all and top 5% SNPs (vertical red dashed line) alongside the expected null distribution of ΔF (solid black line) derived from permutations. Higher ΔF means a higher increase in clustering with high‐F _ST loci (Figure 2). High ΔF in the observed data relative to the null distribution implies that the null hypothesis that clustering increases no more than expected by chance alone when taking high‐F _ST loci can be rejected, as correctly seen only in the high gene flow with local adaptation scenario.

FIGURE 4

Reduction in high‐grading bias on population structure analyses when using statistically identified outlier SNPs. We used simulated data of three types of population structure: (A) panmictic (without selection), (B) high gene flow (without selection), and (C) high gene flow with local adaptation (with selection). Panels A–C illustrate PCAs using all SNPs, D–F use only SNPs that were outliers found by PCAdapt, and G‐I illustrate PCAs when only OutFLANK outliers were used. There were no shared SNPs between OutFLANK and PCAdapt; plots with no outlier loci identified are marked. Note that for the bottom‐right plot, the small number of observed outliers from OutFLANK caused many points to be plotted a top of one another in the two‐dimensional PCA, making interpretation challenging; the distribution of PC1 and PC2 scores per population is clearer and thus shown instead.

FIGURE 5

Management implications of high‐grading bias in five sub‐populations of pink salmon ( Oncorhynchus gorbuscha ). We use our R package PCAssess to plot PCAs using all available SNPs (left) and the top 5% SNPs with the highest F _ST values among the population (right). We automate permutation testing to detect high‐grading bias in the package PCAssess (bottom), which shows that the chosen subset of loci does not provide a statistically significant, biologically relevant increase in population structure and thus cannot reject the null hypothesis of high‐grading bias (p > 0.57).