Recombination estimation under complex evolutionary models with the coalescent composite-likelihood method

Abstract

The composite-likelihood estimator (CLE) of the population recombination rate considers only sites with exactly two alleles under a finite-sites mutation model (McVean, G. A. T., P. Awadalla, and P. Fearnhead. 2002. A coalescent-based method for detecting and estimating recombination from gene sequences. Genetics 160:1231-1241). While in such a model the identity of alleles is not considered, the CLE has been shown to be robust to minor misspecification of the underlying mutational model. However, there are many situations where the putative mutation and demographic history can be quite complex. One good example is rapidly evolving pathogens, like HIV-1. First we evaluated the performance of the CLE and the likelihood permutation test (LPT) under more complex, realistic models, including a general time reversible (GTR) substitution model, rate heterogeneity among sites (Gamma), positive selection, population growth, population structure, and noncontemporaneous sampling. Second, we relaxed some of the assumptions of the CLE allowing for a four-allele, GTR + Gamma model in an attempt to use the data more efficiently. Through simulations and the analysis of real data, we concluded that the CLE is robust to severe misspecifications of the substitution model, but underestimates the recombination rate in the presence of exponential growth, population mixture, selection, or noncontemporaneous sampling. In such cases, the use of more complex models slightly increases performance in some occasions, especially in the case of the LPT. Thus, our results provide for a more robust application of the estimation of recombination rates.

Fig. 1

Relative power and false-positive rate of the LPT. Performance of the LPT under the JC2 and JCall is compared to the best results obtained by any of the 14 recombination detection methods evaluated by Posada and Crandall (2001b). (A) Power to detect recombination when θ = 0.01 (B) Power to detect recombination when θ = 0.05 (C) False-positive rate when θ = 0.01 (D) False-positive rate when θ = 0.05. Gamma shape is the shape of the gamma distribution for rate variation among sites (inf = infinity).