Opposing dominance within a color gene block is associated with maladaptive hybrid signal discordance

The divergence of plumage color genes is increasingly recognized as important for speciation in songbirds through its influence on social signaling. However, the behavioral mechanisms underlying the eco-evolutionary feedback that acts across species boundaries is poorly understood. The hybrid zone between Setophaga occidentalis (SOCC) and S. townsendi (STOW) in the Cascade Mountain range provides a natural observatory to test the interplay between genetics, plumage signals, and territoriality in maintaining nascent species boundaries. Recently, we found that selection within a gene block underpinning color variation (ASIP-RALY) has maintained a stable and narrow hybrid zone. Here we investigated the roles of cheek darkness and flank streaking as plumage signals during simulated territorial intrusion, two melanin-based traits underpinned by ASIP-RALY that reflect opposing dominance of SOCC and STOW alleles. We found that both plumage traits act as honest signals of body size in the territorial sex (i.e., males). The opposing dominance effects of ASIP-RALY resulted in plumage signal discordance for heterozygotes, which in turn was associated with reduced territorial performance, a fitness proxy quantified by vocal and physical responses to a decoy intruder. Taken together, this study highlights a behavioral mechanism underlying selection acting on a simple genetic architecture that shapes species boundary despite gene flow.