Selection against archaic hominin DNA in human regulatory regions

Traces of archaic hominin DNA persist in the human gene pool, but are systematically depleted around genes and other functionally important genomic regions. This suggests that many Neandertal and Denisovan alleles had harmful effects on hybrid fitness. We hypothesized that if some harmful effects were mediated by gene dysregulation in specific tissues, alleles previously flagged as archaic using a conditional random field (CRF) should be depleted from those tissues regulatory enhancers compared to "control" alleles matched for allele frequency and the strength of background selection. By this metric, both Neandertal and Denisovan variation appear depleted from enhancers, particularly enhancers that show pleiotropic activity across tissues. This depletion is driven by young archaic SNPs that the CRF confidently identifies as private to Neandertals or Denisovans; older variants that were likely present in both archaic species are not depleted from enhancers. We found that enhancer pleiotropy is not only a predictor of archaic SNP depletion, but also a predictor of intolerance to new mutations as measured by both phastCons scores and the frequency spectrum of African variation. In other respects, however, the landscape of selection against young archaic alleles appears qualitatively different from the landscape of ordinary purifying selection, suggesting that archaic alleles had a different distribution of fitness effects from ordinary new mutations. Most strikingly, fetal brain and muscle are the tissues most depleted of young archaic variation in their regulatory regions, but only brain enhancers appear commensurately intolerant to new mutations. In contrast, fetal muscle enhancers show no evidence of elevated purifying selection relative to other enhancers. This suggests that epistatic incompatibility between human and archaic alleles is needed to explain the degree of archaic variant depletion from fetal muscle enhancers, perhaps due to divergent selection for higher muscle mass in archaic hominins compared to humans.