Segment Anything for Microscopy

Key results

1. ShownFinetuning the full SAM ViT-H model on LiveCELL improved point-prompted, box-prompted and automatic instance segmentation over the default weights. Prompted results surpassed CellPose, but automatic segmentation did not, and most of the gain came from small training subsets.

   LiveCELL test split, mean segmentation accuracy over matching thresholds 0.5 to 0.95 (step 0.05), baseline CellPose trained on LiveCELL. ViT-B ablation: image encoder updates matter most, full-model finetuning is best. Training fractions of 2, 4 and 5 % already capture most of the improvement. Scores are shown only in Figure 2, not in the text. · Finetuning Segment Anything significantly…; Inference and evaluation; Figure legends

2. ShownA light microscopy generalist trained on pooled public datasets nearly matched dataset-specific specialists and outperformed default SAM on all five held-out imaging settings. Comparisons with the CellPose cyto model were mixed across datasets and prompt types.

   Training data: LiveCELL, TissueNet, DeepBacs, Neurips Cell Segmentation, Nucleus DSB, PlantSeg; specialists were slightly better, mostly on DeepBacs (smallest set). Held-out: CovidIF, HPA, Lizard, mouse-embryo, plantseg-ovules. Against CellPose, sometimes only box prompts won and sometimes every SAM mode won, including automatic. ViT-H; scores only in Figure 3. · Light microscopy model trained on diverse data…; Figure legends

3. ShownAn electron microscopy model trained on CREMI, MitoEM, AxonDeepSeg and PlatyEM improved cell and neurite segmentation but degraded mitochondria. The authors therefore conclude that one EM generalist is not feasible with the current SAM architecture and training.

   In-domain test splits: gains for cells, neurites and nuclei, loss for mitochondria. Held-out Lucchi (FIBSEM mitochondria), NucMM-Z (low-resolution SEM nuclei), SNEMI (SEM neurites), SpongeEM (FIBSEM cells, organelles): neurites up, mitochondria down, others mixed. Finetuned model often segments the surrounding cell instead of the mitochondrion. No baseline method compared. · Electron microscopy model improves segmentation…; Figure legends

4. SuggestedIn brightfield organoid annotation, every semi-automatic approach beat manual labeling. Starting from automatic instance segmentation, micro_sam was faster than CellPose 2 before finetuning, and the two performed comparably after in-the-loop finetuning.

   Internal dataset with 20 ground-truth images; per-object time for manual, CellPose (cyto, three images annotated then finetuned twice) and micro_sam (default ViT-H, with/without automatic segmentation, then finetuned for 10 thousand iterations). Finetuning gave a significant gain for single-point prompts, which drive automatic segmentation. Times only in Figure 5. · Interactive annotation tools based on Segment…; Training; Figure legends

5. SuggestedAnnotating nuclei in 3d fruit fly larva EM was almost twice as fast with micro_sam as with ilastik carving, and the masks fit better. For HeLa tracking, micro_sam was slightly faster than Trackmate with CellPose and gave clearly better tracking quality.

   3d: internal volume EM blocks; carving masks misfit some nuclei because its superpixels cannot be corrected; user-data finetuning improved scores in most settings. Tracking: Cell Tracking Challenge HeLa label-free data with added full segmentation ground truth, time per track and challenge metric; Trackmate is fast and accurate only from ground-truth masks. · Interactive annotation tools based on Segment…; Figure legends