Maximum likelihood estimation of species trees from gene trees in the presence of ancestral population structure

Abstract

Though large multilocus genomic datasets have led to overall improvements in phylogenetic inference, they have posed the new challenge of addressing conflicting signals across the genome. In particular, ancestral population structure, which has been uncovered in a number of diverse species, can skew gene tree frequencies, thereby hindering the performance of species tree estimators. Here we develop a novel maximum likelihood method, termed TASTI, that can infer phylogenies under such scenarios, and find that it has increasing accuracy with increasing numbers of input gene trees, contrasting with the relatively poor performances of methods not tailored for ancestral structure. Moreover, we propose a supertree approach that allows TASTI to scale computationally with increasing numbers of input taxa. We use genetic simulations to assess TASTIs performance in the four-taxon setting, and demonstrate the application of TASTI on a six-species Afrotropical mosquito dataset. Finally, we have implemented TASTI in an open-source software package for ease of use by the scientific community.