Covariance Nonstationarity is Evident in Spatial Transcriptomics and Provides a New Categorization of Spatially Varying Genes

Abstract

Gaussian process models underlie many spatial transcriptomics tools but typically assume stationary covariance. To test this assumption, we use approximate Bayes factors from R-INLA to compare stationary and non-stationary Matern covariance functions across 12 Visium datasets; covariance non-stationarity is substantially evident as it appears in 3% to 50% of genes across datasets. We find that gene sets associated with effector functions are enriched among genes favoring non-stationary spatial covariance.