Genealogies and weak purifying selection

Abstract

The assumption that selection alters the genealogical tree of a sample of alleles from a population relative to the neutral expectation underlies several "tests of neutrality." Two recent papers have studied the effect of purifying selection; their suggestive but incomplete results indicate that, in the single site case, the shape of a gene genealogy for a locus may differ only from the neutral expectation. We verify this finding for weak selection using the "ancestral selection graph." We consider a wider range of models, including both a four-allele single-site model and an infinite-sites model. Our results confirm the previous claim for the symmetric-mutation single site model. We emphasize, however, that a neutral-seeming genealogy is consistent with detectable effects of selection on the distribution of allele frequences within the sample. With selection operating, the information about a sample cannot be reduced to the genealogy. As a result, a distinction needs to be made between the selected sites themselves, for which the genealogy offers insufficient information, and linked neutral variation. This distinction seems to have been overlooked in previous papers, yet it has significant implications for the interpretation of data on DNA sequence variation. In particular, it predicts that under purifying selection, the frequency spectrum of neutral mutations will not reflect the skew toward rare polymorphisms at replacement sites even if there is no recombination between them. We caution, however, that the effect of weak selection on the genealogy is specific to the model; a (more realistic) model of multiple linked sites could lead to a more distorted genealogy than is observed for a single site.