RNA2seg: a generalist model for cell segmentation in image-based spatial transcriptomics

Abstract

Imaging-based spatial transcriptomics (IST) enables high-resolution spatial mapping of RNA species. A key challenge in IST is accurate cell segmentation to assign each RNA molecule to the right cell. Here, we present RNA2seg, a novel segmentation algorithm trained on over 4 million cells from MERFISH and CosMx datasets across seven organs using a teacher-student training scheme. RNA2seg integrates RNA point clouds and all available membrane and nuclear stainings. Validation on manually annotated data shows superior performance including in zero-shot and few-shot settings. The method is available as a documented pip package: https://github.com/fish-quant/rna2seg.