Deep Disentangled Representation Learning Reveals Neuron Subtype-Specific Nuclear Morphologies Across Aging in Mice and Humans

We present a computational pipeline that links nuclear morphology to mRNA expression-based cell phenotypes under diverse biological conditions, including aging, disease progression, and drug response, using RNAscope imaging. The pipeline consists of three components: nuclear segmentation from RNAscope images, nuclear morphology identification, and downstream statistical analysis. Central to our approach is a novel unsupervised method, based on deep disentangled representation learning, which effectively captures diverse nuclear morphologies in large-scale datasets, as validated on synthetic benchmarks. We applied the full pipeline to RNAscope data targeting dopaminergic and glutamatergic neuron populations in the midbrains of mice and humans. Our analyses uncovered distinct nuclear morphology differences between dopaminergic and non-dopaminergic, as well as glutamatergic and non-glutamatergic neurons, in both species. Moreover, we identified a significant interaction between neurotransmitter identity and healthy aging in mice, reflected in systematic changes in nuclear morphology. These findings position nuclear morphology as a scalable and informative imaging-based readout of cell identity and physiological state.