High resolution analysis of population structure using rare variants

Various statistical methods have been developed to identify population structure from genetic data, including F-statistics, which measure the average correlation in allele frequency differences between two pairs of populations. However, the SNPs analyzed with F-statistics are often limited to those found as part of microarrays or, in the case of ancient DNA, to SNP capture panels, which are those within the common allele frequency band. Recent advances in sequencing technology increasingly allow generating whole-genome sequencing data, both ancient and modern, which not only enable querying nearly every base of the genome, but also contain numerous rare variants. Rare variants, with their more population-specific distribution, allow detection of population structure with much finer resolution than common variants - an opportunity that has so far been under-exploited. Here, we develop a new statistical method, RAS (Rare Allele Sharing), for summarizing rare allele frequency correlations, similar to F-statistics but with flexible ascertainment on allele frequencies. We test RAS on both published and simulated data and find that RAS has better resolution in distinguishing populations, with appropriate ascertainment. Leveraging this, we further develop the use of RAS to compute ancestry proportions with higher accuracy than existing methods, in cases of closely-related source populations. We implemented the new statistical methods as an R package and a command line tool. In summary, our method can provide new perspectives to identify and model population structure, allowing us to understand more subtle relationships among populations in the recent human past.