Modern humans did not admix with Neanderthals during their range expansion into Europe

Abstract

The process by which the Neanderthals were replaced by modern humans between 42,000 and 30,000 before present is still intriguing. Although no Neanderthal mitochondrial DNA (mtDNA) lineage is found to date among several thousands of Europeans and in seven early modern Europeans, interbreeding rates as high as 25% could not be excluded between the two subspecies. In this study, we introduce a realistic model of the range expansion of early modern humans into Europe, and of their competition and potential admixture with local Neanderthals. Under this scenario, which explicitly models the dynamics of Neanderthals' replacement, we estimate that maximum interbreeding rates between the two populations should have been smaller than 0.1%. We indeed show that the absence of Neanderthal mtDNA sequences in Europe is compatible with at most 120 admixture events between the two populations despite a likely cohabitation time of more than 12,000 y. This extremely low number strongly suggests an almost complete sterility between Neanderthal females and modern human males, implying that the two populations were probably distinct biological species.

Figure 1. Different Models of the Interactions between Neanderthals and Modern Humans

(A) Model of instantaneous mixing of unsubdivided Neanderthal and modern human populations. (B) Same as (A), but with an exponential growth of the modern human population having started before the admixture with Neanderthals. (C) Model of a progressive range expansion of modern humans into Europe. This model is spatially explicit, and the modern human population occupies a different range than the Neanderthal population before the admixture. Under this model, admixture is progressive and occurs because modern humans move into the territory of Neanderthals, a territory that shrinks with the advance of modern humans.

Figure 2. Range Expansion of Modern Humans into Europe from the Near East

Simulations begin 1,600 generations ago, with the area of Europe already colonized by Neanderthals shown in light gray, and an origin of modern human expansion indicated by a black arrow (lane A). Lanes (B–F) show the progression of the wave of advance of modern humans (dark gray) into Europe at different times before present. The black band at the front of the expansion wave represents the restricted zone of cohabitation between modern humans and Neanderthals.

Figure 3. Expected Proportion of Neanderthal Lineages (in Black) among European Samples under Demographic Scenario A (Table 1) at Different Geographic Locations, for Different Interbreeding Rates

(A) One admixture event on average per 50 demes over the whole period of cohabitation between Neanderthals and modern humans; (B) one admixture event per five demes; (C) one admixture event per two demes; (D) one admixture event per deme.

Figure 4. Likelihood of Different Rates of Interbreeding under the Nine Scenarios Described in Table 1

The horizontal bold dashed line corresponds to 14.7% of the maximum likelihood, defining the upper limit of a 95% confidence interval for the interbreeding rates (see, e.g., Kalbfleisch 1985).