Segmentation Matters: Recognizing the Cell Segmentation Challenge in Spatial Transcriptomics

Abstract

Probe-based in situ hybridization spatial transcriptomics has emerged as a state-of-the-art for neuroscience research. Accurate segmentation of neurons and non-neuronal cells, a critical step for downstream analysis, remains a big challenge. Using human sensory ganglia neurons as an example, we systematically explore this problem. We evaluated multiple automatic segmentation approaches using quantitative performance metrics and downstream results. We show that careful parameter tuning is essential for achieving accurate segmentation; however, even with optimized parameters, different models still yield distinct classes of errors. To mitigate these errors, we propose a manual quality check, which can validate and refine automated segmentation results. As a future direction, integrating multi-modal imaging data with tailored neural networks may further improve segmentation accuracy. In summary, our results argue that each automated segmentation method has distinct strengths, weaknesses, and characteristic error patterns; and that a manual review is necessary.