The probability of monophyly of a sample of gene lineages on a species tree

Abstract

Monophyletic groups-groups that consist of all of the descendants of a most recent common ancestor-arise naturally as a consequence of descent processes that result in meaningful distinctions between organisms. Aspects of monophyly are therefore central to fields that examine and use genealogical descent. In particular, studies in conservation genetics, phylogeography, population genetics, species delimitation, and systematics can all make use of mathematical predictions under evolutionary models about features of monophyly. One important calculation, the probability that a set of gene lineages is monophyletic under a two-species neutral coalescent model, has been used in many studies. Here, we extend this calculation for a species tree model that contains arbitrarily many species. We study the effects of species tree topology and branch lengths on the monophyly probability. These analyses reveal new behavior, including the maintenance of nontrivial monophyly probabilities for gene lineage samples that span multiple species and even for lineages that do not derive from a monophyletic species group. We illustrate the mathematical results using an example application to data from maize and teosinte.

Keywords: coalescent theory; monophyly; phylogeography.

Fig. 1.

Notation for computing monophyly probabilities above a species tree node x. Nodes $x_{LL}$, $x_{LR}$, and $x_R$ are leaves. S lineages appear in blue, C lineages in orange, and M lineages in green. The figure illustrates reciprocal monophyly. Sequentially listing the numbers of S, C, and M lineages as a vector, the outputs of branch x are ${\mathbf{n}}_x^O=\left(\mathrm{0,0,1}\right)$. Inputs are ${\mathbf{n}}_x^L=\left(\mathrm{1,1,0}\right)$ and ${\mathbf{n}}_x^R=\left(\mathrm{2,1,0}\right)$. Farther down the tree, branch $x_L$ has inputs ${\mathbf{n}}_{x_L}^L=\left(\mathrm{1,0,0}\right)$ and ${\mathbf{n}}_{x_L}^R=\left(\mathrm{0,1,0}\right)$. Adopting the convention that leaf inputs enter from the left, branch $x_R$ has inputs ${\mathbf{n}}_{x_R}^L=\left(\mathrm{4,1,0}\right)$ and ${\mathbf{n}}_{x_R}^R=\left(\mathrm{0,0,0}\right)$. Descending one more level—which is only possible for $x_L$—the inputs for branch $x_{LL}$ are ${\mathbf{n}}_{x_{LL}}^L=\left(\mathrm{1,0,0}\right)$ and ${\mathbf{n}}_{x_{LL}}^R=\left(\mathrm{0,0,0}\right)$, and for branch $x_{LR}$, they are ${\mathbf{n}}_{x_{LR}}^L=\left(\mathrm{0,2,0}\right)$ and ${\mathbf{n}}_{x_{LR}}^R=\left(\mathrm{0,0,0}\right)$. Branch widths represent constant population sizes but do not indicate relative magnitudes of these sizes.

Fig. 2.

All cases required for computing combinatorial terms $K_S$ and $K_{SC}$ in monophyly probabilities. (A–G) Cases for monophyly of S (Eq. 4). (H) A case for reciprocal monophyly (Eq. 5). In each panel, lineages coalesce from bottom to top, with the width of a shape corresponding to the number of lineages present. A single lineage is represented by a line, and multiple freely coalescing lineages are represented by shaded polygons with horizontal cross-section proportional to the number of extant lineages. Lineages represented in the same shape or in touching shapes can coalesce with each other. Lineage colors follow Fig. 1.

Fig. 3.

The effect on monophyly probabilities of changing two branch lengths in relation to each other. (A) Model species tree. If the branch length coefficient r is 0, then the tree has a polytomy, and if $r=1$, then the tree reduces to a two-species tree. (B–E) The probabilities of $E_S$ (Eq. 7) for monophyly of S for the tree in A under different scenarios: (B) $\left(S_1,S_2,S_3\right)=\left(\mathrm{2,0,2}\right)$, $\left(C_1,C_2,C_3\right)=\left(\mathrm{2,2,2}\right)$. (C) $\left(S_1,S_2,S_3\right)=\left(\mathrm{2,0,2}\right)$, $\left(C_1,C_2,C_3\right)=\left(\mathrm{0,2,0}\right)$. (D) $\left(S_1,S_2,S_3\right)=\left(\mathrm{2,2,0}\right)$, $\left(C_1,C_2,C_3\right)=\left(\mathrm{2,0,2}\right)$. (E) $\left(S_1,S_2,S_3\right)=\left(\mathrm{2,2,0}\right)$, $\left(C_1,C_2,C_3\right)=\left(\mathrm{2,2,2}\right)$.

Fig. 4.

The effect on monophyly probabilities of pooling lineages from separate species. (A–D) Model species trees. Labels record numbers of input lineages (S in blue, C in orange). (E–J) Probabilities of monophyly events. The trajectories represent species trees with six class-S lineages evenly distributed over one (A), two (B), three (C), and six (D) species. (E) $E_S$ (Eq. 7). (F) $E_C$ (Eq. 8). (G) $E_{SC\prime }$ (Eq. 10). (H) $E_{S\prime C}$ (Eq. 11). (I) $E_{SC}$ (Eq. 9). (J) $E_{S\prime C\prime }$ (Eq. 12).

Fig. 5.

Monophyly frequencies in maize and teosinte. (A) Model species tree. (B) Violin-plot distributions across lineage subsamples of monophyly frequencies for four clades. Means of the observed distributions (excluding outliers for the improved and parviglumis clades) appear as circles and theoretical values appear as triangles. Outliers appear for a single point at frequency $\sim$0.43 in the improved clade and for several points at frequency $>$0.17 in the parviglumis clade, with the cross indicating the mean of the parviglumis outliers (Supporting Information).