StabCell: Stability selection for clustering and marker detection in single-cell RNA sequencing

Abstract

Conventional pipelines for differential expression analysis in single-cell RNA sequencing (scRNA-seq) data first cluster individual cells and then test for differentially expressed genes between the resulting clusters. Using the same data for clustering and testing, however, poses a selective inference problem and can result in overconfidence in differences that may not reflect true biological variation. We introduce StabCell, a stability selection framework which integrates clustering and detection of differentially expressed marker genes. By repeatedly performing clustering and differential expression analysis on complementary pairs of random subsamples, StabCell assesses clustering and marker stability, yielding a stable clustering, cell-specific stability scores, and sets of stable marker genes. In simulations, we demonstrate that StabCell generally maintains empirical per-family error rate (PFER) below prespecified thresholds, selecting fewer false positive marker genes than the conventional approach, especially in cases with low signal-to-noise ratio and low sequencing depth. Simultaneously, StabCell provides competitive marker detection performance compared to alternative approaches. An application to a cell differentiation dataset from induced pluripotent stem cells (IPSCs) to cardiomyocytes illustrates that StabCell can reduce overclustering and prioritize biologically meaningful marker genes, improving the robustness and interpretability of scRNA-seq analyses.