Fine scale prediction of ecological community composition using a two-step sequential machine learning ensemble

Prediction is one the last frontiers in ecology. Indeed, predicting fine scale species composition in natural systems is a complex challenge as multiple abiotic and biotic processes operate simultaneously to determine local species abundances. On the one hand, species intrinsic performance and their tolerance limits to different abiotic pressures modulate species abundances. On the other hand there is growing recognition that species interactions play an equally important role in limiting or promoting such abundances within ecological communities. Here, we present a joint effort between ecologists and data scientists to use data-driven models informed by ecological deterministic processes to predict species abundances using reasonably easy to obtain data. To overcome the classical procedure in ecology of parameterizing complex population models of multiple species interactions and poor predictive power, we followed instead a sequential data-driven modeling approach. We use this framework to predict species abundances over 5 years in a highly diverse annual plant community. Our models show a surprisingly high spatial predictive accuracy (RSE ~ 0.13) using only easy to measure variables in the field, yet such predictive power is lost when temporal dynamics are taken into account. This result suggest that predicting the temporal dimension of our system requires longer time series data. Such data would likely capture additional sources of variability that determine temporal patterns of species abundances. In addition, we show that these data-driven models can also inform back mechanistic models of important missing variables that affect species performance such as particular soil conditions (e.g. carbonate availability in our case). Being able to gain predictive power at fine-scale species composition while maintaining a mechanistic understanding of the underlying processes can be a pivotal tool for conservation, specially given the human induced rapid environmental changes we are experiencing. Here, we document how this objective can be achieved by promoting the interplay between classic modelling approaches in ecology and recently developed data-driven models. Author summaryPrediction is challenging but recently developed machine learning techniques allow to dramatically improve prediction accuracy in several domains. However, these tools are often of little application in ecology due to the complexity of gathering information on the needed explanatory variables, which often comprise not only physical variables such as temperature or soil nutrients, but also information about the complex network of species interactions regulating species abundances. Here we present a two-step sequential modelling framework that overcomes these constraints. We first infer potential species abundances training models just with easily obtained abiotic variables, and then use this outcome to fine-tune the prediction of the realized species abundances when taking into account the rest of the predicted species in the community. Overall, our results show a promising way forward for fine scale prediction in ecology.