Joint estimation of relatedness coefficients and allele frequencies from ancient samples

We develop and test a method to address whether DNA samples sequenced from a group of fossil hominin bone or teeth fragments originate from the same individual or from closely related individuals. Our method assumes low amounts of retrievable DNA, significant levels of sequencing error and contamination from one or more present-day humans. We develop and implement a maximum likelihood method that estimates levels of contamination, sequencing error rates and pairwise relatedness co-efficients in a set of individuals. We assume there is no reference panel for the ancient population to provide allele and haplotype frequencies. Our approach makes use of single nucleotide polymorphisms and does not make assumptions about the underlying demographic model. By artificially mating individual genomes from the 1000 Genomes Project, we determine the numbers of individuals at a given genomic coverage that are required to detect different levels of genetic relatedness with confidence.