Statistical tests of the coalescent model based on the haplotype frequency distribution and the number of segregating sites

Abstract

Several tests of neutral evolution employ the observed number of segregating sites and properties of the haplotype frequency distribution as summary statistics and use simulations to obtain rejection probabilities. Here we develop a "haplotype configuration test" of neutrality (HCT) based on the full haplotype frequency distribution. To enable exact computation of rejection probabilities for small samples, we derive a recursion under the standard coalescent model for the joint distribution of the haplotype frequencies and the number of segregating sites. For larger samples, we consider simulation-based approaches. The utility of the HCT is demonstrated in simulations of alternative models and in application to data from Drosophila melanogaster.

Figure 1.—

Haplotype configuration test rejection probabilities as functions of θ, for s = 10 and n = 7. (A) Probabilities of haplotype configurations, obtained from (17). (B) P-values for the HCT, obtained from (20). For each value of θ at which two configurations have equal probability—that is, when the curves for the configurations cross in A—HCT P-values for the two configurations both experience discontinuities in θ. In B, data for a given configuration are shown with the same color and pattern as used to depict the configuration in A.

Figure 2.—

Power of four tests to reject the standard neutral model at significance level α = 0.05, for three values of s (10, 30, and 75) and three types of alternative models—from top to bottom, recombination, exponential population growth, and island migration. In each plot, n = 30 (in the migration model, the sample was separated into two populations each with sample size 15). Simulation of the null model used procedure 3; simulation of the alternative model used procedure 4 in the case of recombination and appropriately modified versions of procedure 3 in the cases of exponential population growth and island migration (see Demographic models).

Figure 3.—

Power of four tests to reject the standard neutral model at significance level α = 0.05 for a null model including recombination (ρ = 10) and two types of alternative models—exponential population growth (top) and island migration (bottom). Simulation of the null model used procedure 4; simulation of the alternative model used appropriately modified versions of procedure 4. The simulations were otherwise the same as in Figure 2.