The Neanderthal genome included harmful mutations that made the hominids around 40% less reproductively fit than modern humans, according to estimates published in the latest issue of the journal GENETICS. Non-African humans inherited some of this genetic burden when they interbred with Neanderthals, though much of it has been lost over time.

The results suggest that these harmful gene variants continue to reduce the fitness of some populations today. The study also has implications for management of endangered species.

"Neanderthals are fascinating to geneticists because they provide an opportunity to study what happens when two groups of humans evolve independently for a long time - and then come back together," says study leader Kelley Harris, of Stanford University. "Our results suggest that inheriting Neanderthal DNA came at a cost."

Previous studies of DNA extracted from Neanderthal remains revealed that these Eurasian hominids were much more inbred and less genetically diverse than modern humans. For thousands of years, the Neanderthal population size remained small, and mating among close relatives seems to have been common.

Then, 50,000-100,000 years ago, groups of anatomically modern humans left Africa and moved to the homelands of their distant Neanderthal cousins. The two groups interbred, mingling their previously distinct genomes. But though a small fraction of the genome of non-African populations today is Neanderthal, their genetic contribution is uneven. Neanderthal sequences are concentrated in certain parts of the human genome, but missing from other regions.

"Whenever geneticists find a non-random arrangement like that, we look for the evolutionary forces that caused it," says Harris.

Harris and her colleague Rasmus Nielsen (University of California, Berkeley / University of Copenhagen) hypothesized that the force in question was natural selection. In small populations, like the Neanderthals', natural selection is less effective and chance has an outsized influence. This allows weakly harmful mutations to persist, rather than being weeded out over the generations. But once such mutations are introduced back into a larger population, such as modern humans, they would be exposed to the surveillance of natural selection and eventually lost.

To quantify this effect, Harris and Nielsen used computer programs to simulate mutation accumulation during Neanderthal evolution and to estimate how humans were affected by the influx of neanderthal genetic variants. The simulations incorporated data on the mutation rates, genome properties, and population dynamics of hominids.

The results suggest that Neanderthals carried many mutations with mild, but harmful effects. The combined effect of these weak mutations would have made Neanderthals at least 40% less fit than humans in evolutionary terms - that is, they were 40% less likely to reproduce and pass on their genes to the next generation.

Related conclusions were reached in an independent study that used very different methods, led by Ivan Juric at the University of California, Davis. This work is currently being peer reviewed and is available at the pre-publication preprint server bioRxiv.

Harris and Nielsen's simulations also suggest that humans and Neanderthals mixed much more freely than originally thought. Today, Neanderthal sequences make up approximately 2% of the genome in people from non-African populations. But Harris and Nielsen estimate that at the time of interbreeding, closer to 10% of the human migrants' genome would have been Neanderthal. Because there were around ten times more humans than Neanderthals, this number is consistent with the two groups acting as as single population that interbred at random. Recent DNA evidence has confirmed that the Neanderthal contribution to Eurasian genomes was higher in the past.

Although most of the harmful mutations bequeathed by our Neanderthal ancestors would have been lost within a few generations, a small fraction likely persists in people today. Harris and Nielsen estimate that non-Africans may have historically had approximately 1% lower reproductive fitness due to their Neanderthal heritage. This is in spite of the small number of Neanderthal gene variants thought to be beneficial today, including genes related to immunity and skin color.

The results also have implications for conserving endangered species. Many vulnerable populations in fragmented habitats face similar genetic problems to the Neanderthals: inbreeding, low genetic diversity, and accumulation of harmful mutations. One management strategy for overcoming these problems is genetic rescue - improving the health of an inbred population by outcrossing it with other populations.

"Genetic rescue is designed to move gene variants from an outbred population to an inbred population," says Harris. "Our results suggest managers must ensure that this movement only goes one way; otherwise harmful mutations from the inbred population may lower the fitness of the outbred group."