Gene genealogy and variance of interpopulational nucleotide differences

Abstract

A mathematical theory is developed for computing the probability that m genes sampled from one population (species) and n genes sampled from another are derived from l genes that existed at the time of population splitting. The expected time of divergence between the two most closely related genes sampled from two different populations and the time of divergence (coalescence) of all genes sampled are studied by using this theory. It is shown that the time of divergence between the two most closely related genes can be used as an approximate estimate of the time of population splitting (T) only when T identical to t/(2N) is small, where t and N are the number of generations and the effective population size, respectively. The variance of Nei and Li's estimate (d) of the number of net nucleotide differences between two populations is also studied. It is shown that the standard error (Sd) of d is larger than the mean when T is small (T much less than 1). In this case, Sd is reduced considerably by increasing sample size. When T is large (T greater than 1), however, a large proportion of the variance of d is caused by stochastic factors, and increase in the sample size does not help to reduce Sd. To reduce the stochastic variance of d, one must use data from many independent unlinked gene loci.