STANCE: a unified statistical model to detect cell-type-specific spatially variable genes in spatial transcriptomics

A significant challenge in analyzing spatial transcriptomics data is the effective and efficient detection of spatially variable genes (SVGs), whose expression exhibits non-random spatial patterns in tissues. Many SVGs show spatial variation in expression that is highly correlated with cell type categories or compositions, leading to the concept of cell type-specific spatially variable genes (ctSVGs). Existing statistical methods for detecting ctSVGs treat cell type-specific spatial effects as fixed effects when modeling, resulting in a critical issue: the testing results are not invariant to the rotation of spatial coordinates. Additionally, an SVG may display random spatial patterns within a cell type, and a ctSVG may exhibit random spatial patterns from a general perspective, indicating that an SVG does not necessarily have to be a ctSVG, and vice versa. This poses challenges in real analysis when detecting SVGs or ctSVGs. To address these problems, we propose STANCE, a unified statistical model developed to detect both SVG and ctSVGs in spatial transcriptomics. By integrating gene expression, spatial location, and cell type composition through a linear mixed-effect model, STANCE enables the identification of both SVGs and ctSVGs in an initial stage, followed by a second stage test dedicated to ctSVG detection. Its design ensures robustness in complex scenarios and the results are spatial rotation invariant. We demonstrated the performance of STANCE through comprehensive simulations and analyses of three public datasets. The downstream analyses based on ctSVGs detected by STANCE suggest promising future applications of the model in spatial transcriptomics and various areas of genome biology. A software implementation of STANCE is available at https://github.com/Cui-STT-Lab/STANCE.