Inferring the probability of the derived versus the ancestral allelic state at a polymorphic site

Abstract

It is known that the allele ancestral to the variation at a polymorphic nucleotide site cannot be assigned with certainty, and that the most frequently used method to assign the ancestral state - maximum parsimony - is prone to mis-inference. Estimates of counts of sites that have a certain number of copies of the derived allele (the unfolded site frequency spectrum, uSFS) made by parsimony are therefore also biased. We previously developed a maximum likelihood method to estimate the uSFS for a focal species, using information from two outgroups and assuming simple models of nucleotide substitution. Here, we extend this approach to infer the uSFS, allowing multiple outgroups, potentially any phylogenetic tree topology and more complex models of nucleotide substitution. We find, however, that two outgroups and assuming the Kimura 2-parameter model is adequate for uSFS inference in most cases. We show that using parsimony for ancestral state inference at a specific site seriously breaks down in two situations. The first is where the outgroups provide no information about the ancestral state of variation in the focal species. In this case, nucleotide variation will be under-estimated if such sites are removed from the data. The second is where the minor allele in the focal species agrees with the allelic state of the outgroups. In this situation, parsimony tends to over-estimate the probability of the major allele being derived, because it fails to account for the fact that sites with a high frequency of the derived allele tend to be rare in most data sets. We present a method that corrects this deficiency, which is capable of providing unbiased estimates of ancestral state probabilities on a site-by-site basis and the uSFS.