Evolution of mating types in finite populations: the precarious advantage of being rare

Sexually reproducing populations with self-incompatibility bear the cost of limiting potential mates to individuals of a different type. Rare mating types escape this cost since they are unlikely to encounter incompatible partners, leading to the deterministic prediction of continuous invasion by new mutants and an ever increasing number of types. However, rare types are also at an increased risk of being lost by random drift. Calculating the number of mating types that a population can maintain requires consideration of both the deterministic advantages and the stochastic risks. By comparing the relative importance of selection and drift, we show that a population of size N can maintain a maximum of approximately N1/3 mating types for intermediate population sizes while for large N we derive a formal estimate. Although the number of mating types in a population is quite stable, the rare type advantage promotes turnover of types. We derive explicit formulas for both the invasion and turnover probabilities in finite populations.