A classification of structured coalescent processes with migration, conditional on the population pedigree

Abstract

Recent analyses of the effects that organismal genealogies or pedigrees of populations have on times to common ancestry for samples of genetic data are extended to cases of population subdivision and migration. Traditional coalescent models marginalize over pedigrees. A finding of a pedigree effect implies that data analysis and interpretation should not be based on the corresponding traditional coalescent model but rather on a coalescent model obtained by conditioning on the pedigree. We apply a straightforward test based on the distribution of pairwise coalescence times to four previously described scenarios of subdivision and migration. These scenarios are defined by the relative magnitudes of four parameters: the number of the local populations or demes, the deme size, the migration fraction, and the probability that migration can occur at all. We find pedigree effects in three scenarios. In two, the effect is weak if the deme size is large. The one scenario without a pedigree effect corresponds to the well known structured-coalescent model. The one scenario with a persistent pedigree effect even in the limit as the deme size tends to infinity involves long periods without gene flow interrupted by pulses of migration. We illustrate our results using simulations and numerical analysis.