The fractional coalescent

A new approach to the coalescent, the fractional coalescent (f-coalescent), is introduced. Two derivations are presented: first, the f-coalescent is based on an extension of the discrete-time Wright-Fisher model. In this extension, for the population of size N, the probability that two randomly selected individuals have the same parent in the previous generation depends on the variable . Second, the f-coalescent is based on an extension of the discrete-time Canning population model that the variance of the number of offspring is assumed as a random variable which depends on the variable . In the second derivation, the f-coalescent emerges also as a continuous-time semi-Markov process. The additional parameter affects the variability in the patterns of the waiting times; values of < 1 lead to an increase of short time intervals, but allows occasionally for very long time intervals. When = 1, the f-coalescent and Kingmans n-coalescent are equivalent. The mode of the distribution of the time of the most recent common ancestor in the f-coalescent is lower than n-coalescent when the number of sample size increases, and the time which modes happen on that is smaller compare to n-coalescent. Also, this distribution showed that the f-coalescent leads to distributions with heavier tails than the n-coalescent. Also, the probability that n genes descend from m ancestral genes for f-coalescent is derived. The f-coalescent has been implemented in the population genetic model inference software MO_SCPCAPIGRATEC_SCPCAP. Simulation studies suggest that it is possible to infer the correct values from data that was generated with known values. When data is simulated using models with < 1 or for three example datasets (H1N1 influenza, Malaria parasites, Humpback whales), Bayes factor comparisons show an improved model fit of the f-coalescent over the n-coalescent.