Ancestree: unified likelihood inference of ancestral alleles under supplied or inferred genealogies

Abstract

Inferring ancestral states--determining, at each polymorphic site, which allele is ancestral and which derived--underpins many downstream population-genetic analyses, from selection scans and the unfolded site-frequency spectrum to demographic inference. However, no existing tool uniformly supports the full range of relevant inputs: plain variant data or ancestral recombination graphs (ARGs), with or without outgroups, while accommodating poly-allelic and recurrently-mutated sites. Here we present Ancestree, a likelihood-based engine that unifies these inputs within a single framework and returns full posteriors over the four nucleotide states at every site. It runs in three modes: a fixed-tree mode that assumes a single topology across sites and co-infers the per-branch substitution rates by maximum likelihood; an ARG mode that reads a different local tree at each site directly from a supplied ancestral recombination graph; and a local-tree mode that instead samples those local trees from the genotype data via a pairwise-coalescent HMM, needing no pre-existing ARG. On simulated data, the genealogy-based modes (ARG and local-tree) are more accurate and scale better, and remain robust under outgroup configurations that violate the fixed-tree assumption. Outgroups themselves remain difficult to replace: per-site inference accuracy on ingroup-polymorphic sites is markedly limited without them, and improves substantially with a single outgroup. The hardest sites are those fixed for the derived allele within the ingroup, which carry no within-ingroup signal and so need several sufficiently deep outgroups to recover, yet these are also highly informative downstream, carrying the high-frequency divergence signal on which selection and adaptation analyses often depend. Ancestree is available at github.com/Sendrowski/Ancestree.