Single-cell-level condition-related signal estimation with batch effect removal through neural discrete representation learning

Advances in single-cell sequencing techniques and the growing volume of single-cell data have created unprecedented opportunities for uncovering the changes in gene expression patterns induced by perturbations or associated with diseases. However, batch effects and non-linearity in single-cell data make single-cell-level estimation challenging. To address these drawbacks, we developed NDreamer, an approach that combines neural discrete representation learning with counterfactual causal matching. NDreamer can be used to estimate the batch-effect-free and condition-related or perturbation-induced signal-preserved expression data from the raw expressions and then estimate single-cell-level perturbation-induced or condition-related signals. Benchmarked on datasets across platforms, organs, and species, NDreamer robustly outperformed previous single-cell-level perturbation effect estimation methods and batch effect denoising methods. Finally, we applied NDreamer to a large Alzheimers disease cohort and uncovered meaningful gene expression patterns between the dementia patients and health controls.