henhaobio
Sign in

bioRxivBioinformaticsPreprint

CellSAM: A Foundation Model for Cell Segmentation

Uriah Israel, Markus MarksiD, Rohit DilipiD, Qilin LiiD, Changhua YuiD, Emily LaubscheriD, Ahamed Iqbal, Elora Pradhan, Ada Ates, Martin AbtiD, Caitlin Brown, Edward Pao, Shenyi Li, Alexander Pearson-Goulart, Pietro PeronaiD, Georgia GkioxariiD, Ross BarnowskiiD, Yisong YueiD, David Van ValeniD

PDFDOICodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

Cells are a fundamental unit of biological organization, and identifying them in imaging data - cell segmentation - is a critical task for various cellular imaging experiments. While deep learning methods have led to substantial progress on this problem, most models are specialist models that work well for specific domains but cannot be applied across domains or scale well with large amounts of data. In this work, we present CellSAM, a universal model for cell segmentation that generalizes across diverse cellular imaging data. CellSAM builds on top of the Segment Anything Model (SAM) by developing a prompt engineering approach for mask generation. We train an object detector, CellFinder, to automatically detect cells and prompt SAM to generate segmentations. We show that this approach allows a single model to achieve human-level performance for segmenting images of mammalian cells, yeast, and bacteria collected across various imaging modalities. We show that CellSAM has strong zero-shot performance and can be improved with a few examples via few-shot learning. Additionally, we demonstrate how CellSAM can be applied across diverse bioimage analysis workflows. A deployed version of CellSAM is available at https://cellsam.deepcell.org/.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 5 (17 Feb 2025) · CC BY 4.0

Bounding boxes from a transformer detector let the Segment Anything Model segment cells automatically; one generalist model beats generalist Cellpose and matches inter-annotator agreement across mammalian, yeast and bacterial images.

Most deep-learning cell segmenters are specialists tied to one imaging modality or target (Cellpose needs a model zoo; Mesmer's centroid-and-boundary representation suits tissue but not elongated bacteria), and SAM's default grid-of-points prompting fails on cellular images whose densities vary widely. CellSAM replaces manual or grid prompts with detector-generated boxes so a single model can both label and segment diverse cell images automatically.

Key results

  1. ShownA single generalist CellSAM segmented with lower error than generalist Cellpose across all data categories and, in almost every comparison, was equivalent to per-dataset Cellpose specialists; training on all data did not degrade CellSAM, whereas Cellpose degraded.

    F1 error (1-F1) in six categories (Tissue, Cell Culture, H&E, Bacteria, Yeast, Nuclear/DSB): CellSAM-generalist below the internally trained Cellpose-generalist in all, and better than pre-trained cyto3 in almost every comparison; Cellpose degraded as a generalist (Fig. S5), CellSAM did not (Fig. S4). Also held on the NeurIPS challenge split after extra hematology fine-tuning. · 2.3 Benchmarking CellSAM’s performance on…; Figure legends

  2. ShownCellSAM's disagreement with expert annotators was statistically indistinguishable from the disagreement among three experts in four data categories.

    Two-sided t-tests compared CellSAM-vs-human error (1-F1) with human-vs-human error; none were significant, with p=0.90 for bacteria, 0.49 for cell culture, 0.18 for tissue and 0.11 for yeast. The number of images per category is not given in the text. · 2.3 Benchmarking CellSAM’s performance on…; Figure legends

  3. SuggestedBox prompts work far better than point prompts for SAM on cells, but they are not enough alone: ground-truth bounding boxes beat point prompts zero-shot, and after neck fine-tuning, box-prompted CellSAM gives high-quality masks on unseen datasets with no further fine-tuning, which supports fast box-based labeling.

    Box vs point prompting is shown in Fig. S2 (described as significantly higher zero-shot performance); box-prompted masks on unseen data are in Fig. S6. No numbers appear in the main text, and the claim that drawing boxes is faster is asserted, not timed. · 2.2 CellSAM creates masks using box prompts…; 2.3 Benchmarking CellSAM’s performance on…

  4. ShownOn held-out LIVECell, zero-shot accuracy varied by cell line; few-shot fine-tuning recovered adequate performance for some lines (e.g., A172) but not for morphologically distant lines such as SHSY5Y.

    Fine-tuning used ten extra fields of view per cell line (10^2 to 10^3 cells) (Fig. S6, S7); per-line values are not in the text. The introduction says CellSAM substantially outperforms existing zero-shot methods, but the results text does not name comparators. · 2.3 Benchmarking CellSAM’s performance on…; 2.1 Construction of a dataset for general cell…; 1 Introduction

  5. SuggestedThe same generalist model, without dataset-specific fine-tuning, plugged into spatial transcriptomics, live-cell tracking and slice-wise 3D workflows.

    Qualitative demos: Polaris gene assignment on MERFISH (mouse ileum, nuclear plus membrane stain) and seqFISH (cultured macrophages, max-intensity projection of spots); HeLa AMP Kinase reporter after 20mM 2-Deoxy-D-glucose tracked over 60 frames (120 minutes); budding yeast lineages and division counts; u-Segment3D on an epidermal organoid and EASI-FISH lateral hypothalamus. No accuracy metrics. · 2.4 CellSAM unifies biological image analysis…; Figure legends

Methods, data and reproducibility

CellSAM shares SAM's ViT backbone between CellFinder, an Anchor DETR object detector that predicts cell bounding boxes, and SAM's mask decoder, which turns those boxes into instance masks. Training has two stages: ViT plus CellFinder on detection, then the ViT is frozen and the SAM neck is fine-tuned on ground-truth boxes and masks; evaluation uses F1 error (1-F1) against Cellpose variants.

  • Literature-curated 2D corpus: TissueNet, DeepBacs, BriFiSeg, Cellpose, Omnipose, YeastNet, YeaZ and H&E datasets (fluorescence, brightfield, phase contrast, mass cytometry imaging) · mammalian cells, yeast and bacteria (as described)
  • 2018 Kaggle Data Science Bowl (DSB) nuclei, evaluated separately as the Nuclear category
  • Phase400: internally collected phase microscopy images · mammalian · n = eight mammalian cell lines
  • LIVECell, held out for zero-shot and few-shot evaluation, filtered to remove low-quality images
  • NeurIPS Cell Segmentation challenge evaluation split, plus an extra hematology dataset used for fine-tuning
  • Expert annotations for human-vs-human and CellSAM-vs-human comparison (tissue, cell culture, bacteria, yeast) · n = three experts
  • Spatial transcriptomics demos: MERFISH of mouse ileum tissue and seqFISH of cultured macrophages · mouse (MERFISH ileum); not stated for macrophages
  • Live-cell and 3D demos: HeLa cells with an AMP Kinase reporter, a budding yeast lineage movie, an epidermal organoid slice, EASI-FISH of lateral hypothalamus · human (HeLa cell line); budding yeast; others not stated · n = 60 frames over 120 minutes (HeLa)
Methods & tools
Segment Anything Model (SAM), Vision Transformer (ViT), Anchor DETR, CellFinder, Cellpose (cyto3), Polaris, u-Segment3D, Napari, MERFISH, seqFISH, EASI-FISH
Reproducibility gaps
Training details and data preprocessing (Appendix A) are in the supplement, which is not part of this text; no software versions or hardware are given, and the Napari plugin has no link in the text.
Limitations (6)
  • Few-shot fine-tuning did not rescue LIVECell lines whose morphology is far from the training data (e.g., SHSY5Y); the authors suggest bounding-box prompting itself may be the limit. · stated by the authors
  • The authors call extension to 3D essential (the 3D demos aggregate per-slice 2D masks) and note the model does not yet use multiple stains or temporal information from movies. · stated by the authors
  • Labeled data still lack perturbed cells and difficult morphologies such as neurons. · stated by the authors
  • Compute-efficient fine-tuning for new image domains and better prompt engineering are named as open needs. · stated by the authors
  • Quantitative comparisons in the text are against Cellpose variants only (plus the NeurIPS challenge split), on a corpus and splits curated by the authors; other tools the text mentions, such as Mesmer, are not benchmarked, and the main F1 values appear only in figures. · noted in this summary
  • The human-level claim rests on non-significant two-sided t-tests with three annotators and an unstated number of images; failing to detect a difference is not a formal equivalence test. · noted in this summary

Read this if you need one segmentation model for mixed tissue, cell culture, yeast and bacterial images, or want to label cells quickly by drawing boxes for a SAM-based model.

Citations
64

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISVandeLoo '25Pachitariu '25Stringer '20Yang '26Bhat '25Sugawara '23
79% semantic matchBioinformatics
Alexandra D. VandeLoo, Nathan J. Malta, Saahil Sanganeriya+4 more·PLoS ONE·Sep 8, 2025· 10 citations