Coalescent histories on phylogenetic networks and detection of hybridization despite incomplete lineage sorting

Abstract

Analyses of the increasingly available genomic data continue to reveal the extent of hybridization and its role in the evolutionary diversification of various groups of species. We show, through extensive coalescent-based simulations of multilocus data sets on phylogenetic networks, how divergence times before and after hybridization events can result in incomplete lineage sorting with gene tree incongruence signatures identical to those exhibited by hybridization. Evolutionary analysis of such data under the assumption of a species tree model can miss all hybridization events, whereas analysis under the assumption of a species network model would grossly overestimate hybridization events. These issues necessitate a paradigm shift in evolutionary analysis under these scenarios, from a model that assumes a priori a single source of gene tree incongruence to one that integrates multiple sources in a unifying framework. We propose a framework of coalescence within the branches of a phylogenetic network and show how this framework can be used to detect hybridization despite incomplete lineage sorting. We apply the model to simulated data and show that the signature of hybridization can be revealed as long as the interval between the divergence times of the species involved in hybridization is not too small. We reanalyze a data set of 106 loci from 7 in-group Saccharomyces species for which a species tree with no hybridization has been reported in the literature. Our analysis supports the hypothesis that hybridization occurred during the evolution of this group, explaining a large amount of the incongruence in the data. Our findings show that an integrative approach to gene tree incongruence and its reconciliation is needed. Our framework will help in systematically analyzing genomic data for the occurrence of hybridization and elucidating its evolutionary role.

FIGURE 1.

Phylogenetic networks and deep coalescence. Two valid coalescent histories that give rise to gene tree (A,(B,(C,D))) are shown within the branches of a phylogenetic network of four species. The effective population sizes, times t₁ and t₂, as well as the probability γ of a gene being inherited through hybridization, all determine the distribution of gene genealogies in the four species.1

FIGURE 2.

The 26 possible gene tree topologies with four leaves. The numbers within the Newick format description of each tree correspond to the branch numbers in the phylogenetic network of Figure 1.2

FIGURE 3.

The species tree of the yeast data set, as proposed by Rokas et al. a) The single optimal tree under the MDC criterion for the data set. The number of extra lineages resulting from reconciling all 106 gene trees within the branches of this tree is 127. b) The best suboptimal tree under the MDC criterion. The number of extra lineages resulting from reconciling all 106 gene trees within the branches of this tree is 134, which is just seven extra lineages away from the optimal value of 127 achieved by the tree in (a). The number on a branch indicates the number of extra lineages along that branch once all 106 gene trees are reconciled within the branches of the tree.3

FIGURE 4.

Estimated γ ( in Equation (3)) from true gene trees. The variance of the estimates is lower than 0.001 in most cases, and reaches a maximum value of 0.008.4

FIGURE 5.

Estimated γ ( in Equation (3)) from reconstructed gene trees. The variance of the estimates is lower than 0.001 in most cases, and reaches a maximum value of 0.008.

FIGURE 6.

Theoretical distribution of 15 gene tree topologies. White bars: γ = 0; gray bars: γ = 0.3; black bars: γ = 0.5.

FIGURE 7.

Three hybridization scenarios for the yeast data set. Each of the networks requires 69 extra lineages to reconcile all 106 gene trees, and depicts a slightly different hybridization scenario. The number on a branch indicates the number of extra lineages along that branch once all 106 gene trees are reconciled within the branches of the network.