A Cautionary Note on Using STRUCTURE to Detect Hybridization in a Phylogenetic Context

Population genetic clustering methods are widely used to detect hybridization events between closely related populations within species, as well as between deeply diverged lineages across phylogenetic time-scales, although their strengths and limitations in the latter cases remain poorly explored. This study presents the first systematic evaluation of the performance of the most popular population clustering method, STRUCTURE, under a variety of cross-species hybridization scenarios, including hybrid speciation, as well as introgression involving ghost (i.e., extinct or unsampled) lineages or otherwise. Our simulations demonstrate that STRUCTURE performs well in identifying hybrids and their parental donors only when admixture happens very recently between sampled extant lineages. However, STRUCTURE generally fails to detect signals of admixture when hybridization occurs in deep time or when gene flow stems from ghost lineages. We find that symmetrical parental contribution in cases of hybrid speciation will often be revealed as extremely asymmetrical in STRUCTURE, especially when the admixture event occurred more than some time ago. Our results suggest that population-genetic clustering methods may be very inefficient for detecting either ancient or ghost admixtures, partly explaining why ghost introgression has escaped the attention of evolutionary biologists until recently.