Unified cell segmentation for spatial transcriptomics via transcript reassignment

Abstract

High-resolution spatial transcriptomics (ST) measures gene expression as individually localized molecules on imaging-based platforms or as counts in subcellular bins on sequencing-based platforms, and both must be assigned to cells before cell-level analysis. Existing segmentation methods focus on delineating cell boundaries, yet transcripts that diffuse beyond their cell of origin, fall where cells overlap, or lie in bins that cells only partially cover are still misassigned. Here we present Tracel, a two-stage optimal transport framework that assigns spatial observations to cells under the guidance of a matched single-cell RNA-sequencing reference. Tracel first aligns reference cells to spatial cells to estimate the expected expression profile at each cell location, and then maps transcripts to cells against these profiles. The same formulation assigns transcripts from imaging-based platforms (e.g., Xenium, CosMx) and bin counts from sequencing-based platforms (e.g., Visium HD). Across five human datasets spanning three platforms and three tissues, Tracel improves the separation of cell types over existing methods, assigning cell-type markers to their correct cells, retaining more cell-intrinsic transcripts and reducing contamination from neighboring cells. Tracel also preserves variation within cell types, yielding more accurate differential expression across spatial locations, and its expression profiles attribute ligand-receptor interactions to the correct cell-type pairs in cell-cell communication analysis. Tracel can enter the segmentation workflow at different stages, either refining an existing segmentation or segmenting cells starting from nucleus segmentation.