On the robustness to gene tree rooting (or lack thereof) of triplet-based species tree estimation methods

Abstract

Species tree estimation is frequently based on phylogenomic approaches that use multiple genes from throughout the genome. This process becomes particularly challenging due to gene tree heterogeneity (discordance), often resulting from Incomplete Lineage Sorting (ILS). Triplet and quartet-based approaches for species tree estimation have gained substantial attention as they are provably statistically consistent in the presence of ILS. However, unlike quartet-based methods, the limitation of rooted triplet-based methods in handling unrooted gene trees has restricted their adoption in the systematics community. Furthermore, since the induced triplet distribution in a gene tree depends on the placement of the root, the accuracy of triplet-based methods depends on the accuracy of gene tree rooting. Despite progress in developing methods for rooting unrooted gene trees, greatly understudied is the choice of rooting technique and downstream effects on species tree inference under realistic model conditions. This study involves rigorous empirical testing with different gene tree rooting approaches to establish a nuanced understanding of the impact of rooting on species tree accuracy. Moreover, we aim to investigate the conditions under which triplet-based methods provide more accurate species tree estimations than the widely-used quartet-based methods such as ASTRAL.