Spatial Gene Set Enrichment Analysis with Applications to Spatially Resolved Transcriptomic Data

Abstract

Spatially resolved transcriptomics enables the systematic characterization of spatial gene expression variation across tissue sections. Spatially variable genes within the same biological pathway often exhibit similar spatial expression patterns, reflecting shared biological functions and tissue organization. However, existing gene set enrichment analysis methods typically ignore this spatial dependence, which may reduce power to detect spatially organized pathways and limit the interpretability of pathway-level findings. To address this limitation, we propose spaGSE, a Bayesian hierarchical model for spatial pathway enrichment analysis that integrates genelevel summary statistics from spatial expression analysis with predefined gene set annotations. spaGSE models latent spatially variable gene signals through a Gaussian mixture framework and links spatial variation to gene set membership using logistic regression. To support robust and interpretable inference, we impose a spike-and-slab prior on the enrichment coefficient. Through simulation studies and analyses of four public SRT datasets, we show that spaGSE is scalable and achieves higher power while maintaining false positive rate control compared with existing approaches. In real-data applications, spaGSE identifies biologically relevant pathways with coordinated spatial organization across cancer and developmental tissues, demonstrating the value of incorporating spatial information into pathway-level inference for spatial transcriptomics.