Disentangling spatial and environmental effects: flexible methods for community ecology and macroecology

Community ecologists and macroecologists have long sought to evaluate the importance of environmental conditions in determining species distributions, community composition, and diversity across sites. Different methods have been used to estimate species-environment relationships, but their differences to jointly fit and disentangle spatial autocorrelation and structure remain poorly studied. We compared how methods in four broad families of statistical models estimated the contribution of the environment and space to variation in species binary occurrence and abundance. These methods included distance-based regression, generalized linear models (GLM, and the special case of RDA), generalized additive models (GAM), and tree-based machine learning (ML): regression trees, boosted regression trees (BRT), and random forests. The spatial component of the model consisted of spatial distance (in distance-based regression), Morans Eigenvector Maps (MEM; in GLM and ML), smooth spatial splines (in GAM), or tree-based non-linear modelling of spatial coordinates (in ML). We simulated typical site-by-species data to assess the methods performance in (1) fitting environmental and spatial models, and (2) partitioning the variation explained by environmental and spatial predictors. We observed marked differences in performance mostly caused by imbalanced performance in estimating environmental and spatial effects. Such differences also manifested when analyzing eight different empirical datasets. GLM and BRT with MEMs were generally the most reliable methods for partitioning the variation explained by environmental and spatial effects across a wide range of simulated scenarios. The remaining methods tended to underfit simulated spatial structures, causing underestimation of spatial fractions of variation. Our results suggest that previously overlooked methods for performing variation partitioning, especially tree-based ML, offer flexible approaches to analyze site-by-species matrices. We provide general guidelines on the usefulness of different models under different ecological and sampling scenarios, for species distribution modelling, community ecology, and macroecology.