On the Effect of Intralocus Recombination on Triplet-Based Species Tree Estimation

Abstract

We consider species tree estimation from multiple loci subject to intralocus recombination. We focus on R*, a summary coalescent-based methods using rooted triplets. We demonstrate analytically that intralocus recombination gives rise to an inconsistency zone, in which correct inference is not assured even in the limit of infinite amount of data. In addition, we validate and characterize this inconsistency zone through a simulation study that suggests that differential rates of recombination between closely related taxa can amplify the effect of incomplete lineage sorting and contribute to inconsistency.