Gene genealogies within mutant allelic classes

Abstract

A coalescent theory of the gene genealogy within an allelic class that arises by a unique mutational event is developed and analyzed. To interpret this theory it was necessary to expand on existing theory for populations of varying size. Two features of the gene genealogy--the average pairwise distance and the total tree length--within the mutant class and within the nonmutant class are found. An index, I, is proposed that describes the extent to which a genealogy is similar to one from a population of constant size (for which I = 0) or to a star genealogy (for which I = 1). The value of I is positive in growing populations and is generally positive for the gene genealogy for the mutant class. The value of I is negative for a population decreasing in size and for the nonmutant class, if the mutant arose recently. The results are discussed in the context of the infinite sites model of mutation, which is appropriate for nucleotide sequence data, and the generalized stepwise mutation model, which is appropriate for microsatellite loci. The same genealogical methods are used to find the probability of at least one recombination event between the nucleotide that defines an allelic class and a marker at a nearby linked site.