Differential Effects of Incomplete Lineage Sorting and Gene Tree Estimation Error on Gene Tree Distributions and Species Tree Inference

Abstract

Accurate species tree inference from genome-scale data is complicated by gene tree discordance, which can arise both from biological processes such as incomplete lineage sorting (ILS) and from technical factors such as gene tree estimation error (GTEE). While both factors reduce the accuracy of summary methods, their relative impact and characteristic patterns remain poorly understood. Here, we systematically compare the effects of ILS and GTEE by simulating gene tree datasets with comparable overall discordance levels, but with discordance arising exclusively from either ILS or GTEE. Using widely employed summary methods such as ASTRAL and wQFM, we show that GTEE typically has a stronger detrimental effect on species tree accuracy than ILS, even at matched discordance levels. We further characterize the structure of gene tree distributions under these two sources of discordance and show that ILS induces a structured, constrained skew in quartet distributions, whereas GTEE generates more uniform, high-entropy noise that does not diminish with additional genes. Our case study on a widely used avian phylogenomic dataset reveals similar distributional patterns across exons, introns, and ultraconserved elements (UCEs), which differ substantially in their levels of phylogenetic signal. A quartet-based analysis of these gene trees further shows that prioritizing loci with stronger and more consistent quartet support can improve the recovery of established avian clades. Overall, these results provide an empirical framework for a nuanced understanding of how ILS and GTEE shape gene tree distributions and influence species tree inference from limited or noisy gene tree datasets.