Cellpose-SAM: superhuman generalization for cellular segmentation

Key results

1. ShownOn the Cellpose test set, chosen because its test images differ most from the training images, Cellpose-SAM makes fewer errors than a second human annotator and than earlier models, approaching the estimated human-consensus bound.

   Error rate at IoU 0.5 on the Cellpose test set: Cellpose-SAM 0.163, Cellpose3 0.292, CellSAM 0.328; Annotator2 (scored against Annotator1) 0.257; consensus estimate 0.128, half that rate. Training used Annotator1 labels only. On the more homogeneous TissueNet, LiveCell, Omnipose, DeepBacs and MoNuSeg test sets it matched or beat other models, typically by smaller margins (values not in this text). · Model validation; Human-consensus bound; Segmentation benchmarks

2. ShownA single model copes with arbitrary channel order, a wide range of cell sizes without rescaling, and noisy, blurred or downsampled images without a separate restoration step.

   Channel permutation in training (RGB, BRG, GBR and random orders tested) cost no accuracy; trained on diameters 7.5 to 120 pixels and tested at 10 to 90, with a small loss only for small cells; under Poisson noise, blur and (anisotropic) downsampling it matched the authors' best earlier restoration models, each trained for one degradation. Scores are not in this text. · Invariance and robustness; Color and size invariance; Robustness to image degradation as

3. ShownAs an initialisation for fine-tuning on data unseen in training, Cellpose-SAM outperforms cyto3 in 2D and 3D, with the advantage largest when few labelled images are available.

   On BlastoSPIM mouse embryo volumes and PlantSeg lateral root and ovule volumes, using 1 to 512 training images or the full set, Cellpose-SAM started higher and improved equally fast or faster than cyto3; the gap generally narrowed with more data but sometimes persisted. Fine-tuning used batch size 1, which the authors say is always feasible even on 8-12 GB GPUs. Exact scores are not in this text. · Finetuning Cellpose-SAM in 2D and 3D and for…; Fine-tuning and evaluation; Fine-tuning datasets

4. ShownDespite about 50 times more parameters than Cellpose, Cellpose-SAM was the fastest per image among the compared models.

   Speed is attributed to GPU tensor cores accelerating transformer layers, post-processing taking a large share of runtime, and dropping the separate size-estimation pass that Cellpose 1/2/3 need; inputs were reduced from 1024×1024 to 256×256 with 8×8 patches. Timing values are not in this text. · Model design; Model validation

5. ShownRetrained with extra class-output maps, a Cellpose-SAM-based model jointly segments and classifies nuclei and beat the MoNuSAC 2020 challenge winners on nearly every class.

   Four H&E nucleus classes (epithelial, lymphocyte, macrophage, neutrophil); compared against predictions shared by winners PL1, PL2, PL3 and L2 (the L1 link did not work), it was better on nearly all cell types and raised overall classification accuracy substantially. Per-class error rate and average precision values are not in this text. · Finetuning Cellpose-SAM in 2D and 3D and for…; Semantic segmentation training and evaluation