STAREG: an empirical Bayesian approach to detect replicable spatially variable genes in spatial transcriptomic studies

Abstract

Identifying replicable genes that display spatial expression patterns from different yet related spatially resolved transcriptomic studies provides stronger scientific evidence and more powerful inference. We present an empirical Bayesian method, STAREG, for identifying replicable spatially variable genes in data generated from various spatially resolved transcriptomic techniques. STAREG models the joint distribution of p-values from different studies with a mixture model and accounts for the heterogeneity of different studies. It provides effective control of the false discovery rate and has higher power by borrowing information across genes and different studies. Moreover, it provides different rankings of important spatially variable genes. With the EM algorithm in combination with pool-adjacent-violator-algorithm (PAVA), STAREG is scalable to datasets with tens of thousands of genes measured on tens of thousands of spatial spots without any tuning parameters. Analyzing three pairs of spatially resolved transcriptomic datasets using STAREG, we show that it makes biological discoveries that otherwise cannot be obtained by using existing methods.