A hybrid machine learning and regression method for cell type deconvolution of spatial barcoding-based transcriptomic data

Spatial barcoding-based transcriptomic (ST) data require cell type deconvolution for cellular-level downstream analysis. Here we present SDePER, a hybrid machine learning and regression method, to deconvolve ST data using reference single-cell RNA sequencing (scRNA-seq) data. SDePER uses a machine learning approach to remove the systematic difference between ST and scRNA-seq data (platform effects) explicitly and efficiently to ensure the linear relationship between ST data and cell type-specific expression profile. It also considers sparsity of cell types per capture spot and across-spots spatial correlation in cell type compositions. Based on the estimated cell type proportions, SDePER imputes cell type compositions and gene expression at unmeasured locations in a tissue map with enhanced resolution. Applications to coarse-grained simulated data and four real datasets showed that SDePER achieved more accurate and robust results than existing methods, suggesting the importance of considering platform effects, sparsity and spatial correlation in cell type deconvolution.