Fast and accurate joint inference of coancestry parameters for populations and/or individuals

We introduce a fast, new algorithm for inferring jointly the FST parameters describing genetic distances among a set of populations and/or unrelated diploid individuals, and a tree representing their genetic structure, from allele count data. While the inferred tree typically reflects historical processes of splitting and divergence, its aim is to represent the actual genetic variance, with FST values specified by branch lengths. We generalise two major approaches to defining FST, via correlations and mismatch probabilities of sampled allele pairs, which measure shared and non-shared components of genetic variance. A diploid individual can be treated as a population of two gametes, which allows inference of coancestry coefficients for individuals as well as for populations, or a combination of the two. A simulation study illustrates that our fast method-of-moments estimation of FST values, simultaneously for multiple populations/individuals, gains statistical efficiency over pairwise approaches by pooling information about ancestral allele frequencies. We apply our approach to genome-wide genotypes from the 26 worldwide human populations of the 1000 Genomes Project. We first analyse at the population level, then a subset of individuals and in a final analysis we pool individuals from the more homogeneous populations. This flexible analysis approach gives many advantages over traditional approaches to population structure/coancestry, including visual and quantitative assessments of long-standing questions about the relative magnitudes of within- and between-population genetic differences. Author summaryWe propose new ways to measure, and visualise in a tree, the genetic distances among a set of populations using allele frequency data. The two genomes within a diploid individual can be treated as a small population, which allows a flexible framework for investigating genetic variation within and between populations. Genetic structure can be accurately and efficiently represented in a tree with nodes representing either homogeneous populations or genetically diverse individuals, for example due to admixture. We first generalise the long-established measure of genetic distance, FST, to tree-structured populations and individuals, finding that two measures are required for each pair of populations, corresponding to their shared and and non-shared genetic variation. We show using a simulation study that our novel tree-based estimators are more efficient than current pairwise estimators, and we illustrate the potential for novel ways to explore and visualise genetic variation within and between populations using a worldwide human genetic dataset.