Inference from gene trees in a subdivided population

Abstract

This paper studies gene trees in subdivided populations which are constructed as perfect phylogenies from the pattern of mutations in a sample of DNA sequences and presents a new recursion for the probability distribution of such gene trees. The underlying evolutionary model is the coalescent process in a subdivided population. The infinitely-many-sites model of mutation is assumed. Ancestral inference questions that are discussed are maximum likelihood estimation of migration and mutation rates; detection of population growth by likelihood techniques; determining the distribution of the time to the most recent common ancestor of a sample of sequences; determining the distribution of the age of the mutations on the gene tree; determining in which subpopulation the most recent common ancestor of all the sequences was; determining subpopulation ancestors, where they were, and times to them; and determining in which subpopulations mutations occurred. A computational technique of Griffiths and Tavaré used is a computer intensive Markov chain simulation, which simulates gene trees conditional on their topology implied by the mutation pattern in the sample of DNA sequences. The software GENETREE, which implements these ancestral inference techniques, is available.