Adaptive Cluster-Count Autoencoders with Dirichlet Process Priors for Geometry-Aware Single-Cell Representation Learning

Abstract

Standard autoencoders for single-cell transcriptomics learn latent spaces whose cluster structure emerges only post hoc through K-means or community detection, leaving cluster count and boundary quality uncontrolled during training. Here we ask whether imposing an adaptive nonparametric prior can shift this balance. We equip a feedforward autoen-coder with an online Dirichlet Process Mixture Model (DPMM) prior that refits cluster assignments throughout training and directly regularizes latent compactness and separation. Across 56 scRNA-seq datasets the DPMM prior produces a pronounced geometry- concordance trade-off : cluster compactness (ASW) improves by 127% and Davies-Bouldin overlap drops by 47%, but label-recovery metrics decline (NMI -17%, ARI -21%) and downstream kNN accuracy falls from 0.784 to 0.725. Wilcoxon signed-rank tests confirm that the geometry gains are significant with large Cliffs{delta} effects while concordance losses remain bounded and non-significant. A second-stage conditional-flow refinement (DPMM-FM) further improves projection fidelity (DRE 0.751, LSE 0.695, DREX 0.873) at additional concordance cost, revealing a three-tier operating regime: prior-free for label recovery, DPMM for manifold geometry, and DPMM-FM for visualization fidelity. Against 18 external baselines DPMM-Base wins 70.5% of core-metric comparisons (p<0.05). Gene Ontology enrichment confirms that geometry-improved latent components recover coherent biological programs. Rather than claiming universal superiority, this study characterizes the operating envelope of nonparametric mixture priors and identifies the task contexts-- trajectory analysis, manifold visualization, and program-level annotation--where adaptive geometric structure outweighs label-counting accuracy.