Robust detection of natural selection using a probabilistic model of tree imbalance

Neutrality tests such as Tajimas D (Tajima, 1989) and Fay and Wus H (Fay and Wu, 2000) are standard implements in the population genetics toolbox. One of their most common uses is to scan the genome for signals of natural selection. However, it is well understood that deviance measures like D and H are confounded by other evolutionary forces--in particular, population expansion--that may be unrelated to selection. Because they are not model-based, it is not clear how to deconfound these statistics in a principled way. In this paper we derive new likelihood-based methods for detecting natural selection which are robust to confounding by fluctuations in effective population size. At the core of our method is a novel proba-bilistic model of tree imbalance, which generalizes Kingmans coales-cent to allow certain aberrant tree topologies to arise more frequently than is expected under neutrality. We derive a frequency spectrum-based estimator which can be used in place of D, and also extend to the case where genealogies are first estimated. We benchmark our meth-ods on real and simulated data, and provide an open source software implementation.