Effect of ancient population structure on the degree of polymorphism shared between modern human populations and ancient hominins

Abstract

Recent comparisons between anatomically modern humans and ancient genomes of other hominins have raised the tantalizing, and hotly debated, possibility of hybridization. Although several tests of hybridization have been devised, they all rely on the degree to which different modern populations share genetic polymorphisms with the ancient genomes of other hominins. However, spatial population structure is expected to generate genetic patterns similar to those that might be attributed to hybridization. To investigate this problem, we take Neanderthals as a case study, and build a spatially explicit model of the shared history of anatomically modern humans and this hominin. We show that the excess polymorphism shared between Eurasians and Neanderthals is compatible with scenarios in which no hybridization occurred, and is strongly linked to the strength of population structure in ancient populations. Thus, we recommend caution in inferring admixture from geographic patterns of shared polymorphisms, and argue that future attempts to investigate ancient hybridization between humans and other hominins should explicitly account for population structure.

Fig. 1.

Schematic representation of the spatially structured model representing the history of Neanderthals and anatomically modern humans. We start with a connected string of demes representing the common ancestor (in light blue), spanning Africa and Eurasia (A). Following the split (B), the northern part of the range becomes Neanderthals (purple), and the African part of the range eventually becomes modern humans (in orange). From this African range, modern humans later expand to colonize Eurasia (in red) (C). This figure only provides the logic of the model; for details, including the actual number of demes, see Materials and Methods.

Fig. 2.

Observed and simulated D statistics. D statistics for Europe vs. Africa (red), Asia vs. Africa (blue), and Europe vs. Asia (orange) in the original data (22) (triangles, with a 95% confidence interval) and as predicted by our spatial model (lines, representing the distributions of possible values).