The probability of a gene tree topology within a phylogenetic network with applications to hybridization detection

Abstract

Gene tree topologies have proven a powerful data source for various tasks, including species tree inference and species delimitation. Consequently, methods for computing probabilities of gene trees within species trees have been developed and widely used in probabilistic inference frameworks. All these methods assume an underlying multispecies coalescent model. However, when reticulate evolutionary events such as hybridization occur, these methods are inadequate, as they do not account for such events. Methods that account for both hybridization and deep coalescence in computing the probability of a gene tree topology currently exist for very limited cases. However, no such methods exist for general cases, owing primarily to the fact that it is currently unknown how to compute the probability of a gene tree topology within the branches of a phylogenetic network. Here we present a novel method for computing the probability of gene tree topologies on phylogenetic networks and demonstrate its application to the inference of hybridization in the presence of incomplete lineage sorting. We reanalyze a Saccharomyces species data set for which multiple analyses had converged on a species tree candidate. Using our method, though, we show that an evolutionary hypothesis involving hybridization in this group has better support than one of strict divergence. A similar reanalysis on a group of three Drosophila species shows that the data is consistent with hybridization. Further, using extensive simulation studies, we demonstrate the power of gene tree topologies at obtaining accurate estimates of branch lengths and hybridization probabilities of a given phylogenetic network. Finally, we discuss identifiability issues with detecting hybridization, particularly in cases that involve extinction or incomplete sampling of taxa.

Figure 1. Phylogenetic networks, MUL trees, and valid allele mappings.

In this example, single alleles , , and were sampled from each of the three species , , and , respectively, whereas two alleles ( and ) were sampled from species . See text and Text S1 for details.

Figure 2. Various hypotheses for the evolutionary history of a yeast data set.

(A) The species tree for the five species Sbay, Skud, Smik, Scer, and Spar, as proposed in , and inferred using a Bayesian approach and a parsimony approach . (B) A slightly suboptimal tree for the five species, as identified in , . (C–E) The three phylogenetic networks that reconcile both trees in (A) and (B), and which we reported as equally optimal evolutionary histories under a parsimony criterion in . (F) A phylogenetic network that postulates Smik and Skud as two sister taxa whose divergence followed a hybridization event.

Figure 3. Six hypotheses for the evolutionary history of a Drosophila data set.

(A–C) The three possible species tree topologies. (D–E) The three possible single-hybridization species network topologies (excluding extinction events).

Figure 4. Identifiability in detecting hybridization.

(A) A phylogenetic network with two hybridization probabilities, where the second hybridization involves the first hybrid population, and extinction is involved. (B–D) Estimates of and , as a function of the number of gene trees used, when the true values of are assumed in the inference, and for true values of , , and , respectively (insets zoom in on the left parts of the figure). (E) A phylogenetic tree with three taxa, and with divergence time between the two speciation events. (F–H) The value of for the tree in (E) that yields the same probability of the data under the scenario depicted in (A) when , as a function of and , and for value of , , and , respectively. Since a single allele was sampled per species, the data is uninformative for estimating the value of here.