Caliban: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning
Morgan Sarah SchwartziD, Erick MoeniD, Geneva MilleriD, Tom DoughertyiD, Enrico Borba, Rachel Ding, William GrafiD, Edward Pao, David Van ValeniD
Abstract
While live-cell imaging is a powerful approach to studying the dynamics of cellular systems, converting these imaging data into quantitative, single-cell records of cellular behavior has been a longstanding challenge. Deep learning methods have proven capable of performing cell segmentation--a critical task for analyzing live-cell imaging data--but their performance in cell tracking has been limited by a lack of dynamic datasets with temporally consistent single-cell labels. We bridge this gap through the integrated development of labeling and deep learning methodology. We present a new framework for scalable, human-in-the-loop labeling of live-cell imaging movies, which we use to label a large collection of movies of fluorescently labeled cell nuclei. We use these data to create a new deep-learning-based cell-tracking method that achieves state-of-the-art performance in cell tracking. We have made all of the data, code, and software publicly available with permissive open-source licensing through the DeepCell projects web portal https://deepcell.org.
02 /Specification & Classification
Click any tag to browse matching papers- Citations
- 74
- Organism
- Assay
- Research Focus
- Approach
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 5 (7 Aug 2024) · CC BY 4.0
Human-in-the-loop labeling of nuclear live-cell movies produced DynamicNuclearNet (over 2,621 divisions); Caliban, a tracker trained on it, beat four published trackers at division detection.
Deep learning has advanced cell segmentation, but tracking has lagged because existing dynamic datasets are limited in label resolution, trajectory length and number of mitoses, and simulated data have not yet yielded high-performing models. Extending human-in-the-loop labeling from static images to movies, the work builds a pixel-level nuclear tracking set with more division events than earlier mask-based sets and trains a division-aware tracker on it.
Key results
ShownDynamicNuclearNet, built by two-phase human-in-the-loop labeling, holds pixel-level nuclear masks from five cell lines, linked into trajectories with annotated divisions. It has more division events than any earlier tracking dataset with nuclear masks listed by the authors.
647,322 unique nuclear segmentations, over 16,501 trajectories and over 2,621 divisions, from movies of 42-92 frames; compared with public 2D+T tracking datasets in Table 1. Four annotators did the labeling, and one of two expert reviewers approved each movie. · 1.1 HITL labeling of large-scale, dynamic…; 3.3.2 Data annotation; Tables
ShownOn the DynamicNuclearNet test split, Caliban beat Baxter, CellTrackerGNN, EmbedTrack and Trac^x on division metrics. The gain held on ground-truth segmentations, which points to the tracking model rather than segmentation. On segmentation and linkage metrics it was comparable.
Division recall, precision, F1 and mitotic branching correctness (MBC), plus Cell Tracking Challenge (CTC) DET, SEG and TRA, association accuracy and target efficiency, are in Supplementary Table 1 and Fig. 2d; the text states no values. Each method ran on ground-truth and on predicted segmentations (EmbedTrack predicted only). · 1.2 Accurate nuclear segmentation and tracking…; 3.7 Benchmarking; Figure legends; Tables
SuggestedOn the two Cell Tracking Challenge Fluo-N2DL-HeLa test movies, Caliban outperformed the four published methods on every metric reported.
Values are in Supplementary Table 2; the text gives none. The authors annotated the test movies themselves and have not released the labels. Caliban's single model was trained on all five DynamicNuclearNet cell types, including both Fluo-N2DL-HeLa training movies. Comparators used existing models or parameters tuned on Fluo-N2DL-HeLa where available. · 1.2 Accurate nuclear segmentation and tracking…; 3.7 Benchmarking; Tables
ShownSegmentation remains a performance bottleneck: tracking methods do worse on deep-learning-generated segmentations than on ground-truth masks.
The authors cite the gap between tracking on ground-truth and on predicted segmentations in Fig. 2d; the text gives no values. The segmentation model (a feature pyramid network on an EfficientNetV2L backbone) was chosen after testing ten backbones. · 2 Discussion; 3.4.1 Deep learning model architecture; 3.4.5 Model training and optimization
ShownCaliban runs fast enough for cloud deployment in the DeepCell Kiosk; the authors credit this to splitting tracking into a GPU-parallel neighborhood encoder and a lightweight frame-by-frame tracking step.
About 40 s for a typical dataset of 10,000 cell detections over 30 frames on one NVIDIA RTX A6000 GPU with eight AMD EPYC 7763 CPUs, mostly spent on segmentation (Supplementary Fig. 1). Each test-split movie was run three times. · 2 Discussion; 3.7 Benchmarking; 1.2 Accurate nuclear segmentation and tracking…; 3.6 Deployment
Methods, data and reproducibility
Nuclear movies were labeled in DeepCell Label by repeated rounds of model prediction, crowdsourced correction, expert review and retraining, first for segmentation, then tracking. Caliban segments frames with a feature pyramid network, embeds cells with a graph attention network, scores cell pairs as same, different or parent-child using long short-term memory layers, and links them by Hungarian assignment.
- DynamicNuclearNet: time-lapse movies of H2B-tagged nuclei from NIH-3T3, HeLa-S3, HEK293, RAW 264.7 and PC-3 cells, segmented and tracked; 70/20/10 train/validation/test split · mammalian; human and mouse by cell-line identity · n = 647,322 segmentations, over 16,501 trajectories, over 2,621 divisions
- Cell Tracking Challenge Fluo-N2DL-HeLa training movies, given complete segmentation masks and added to DynamicNuclearNet · human (HeLa cell line) · n = 2 training movies
- Cell Tracking Challenge Fluo-N2DL-HeLa test movies, manually annotated by the authors for benchmarking (annotations not released) · human (HeLa cell line) · n = 2 test movies
- Methods & tools
- DeepCell Label, Data Version Control (DVC), XState, deepcell-tf, deepcell-tracking, DeepCell Kiosk, EfficientNetV2L feature pyramid network, Baxter, CellTrackerGNN, EmbedTrack, Trac^x, Cell Tracking Challenge evaluation software
- Code
- https://github.com/vanvalenlab/deepcell-tfhttps://github.com/vanvalenlab/deepcell-trackinghttps://github.com/vanvalenlab/deepcell-labelhttps://github.com/vanvalenlab/kiosk-consolehttps://github.com/vanvalenlab/Caliban-2024_Schwartz_et_al
- Data & accessions
- https://deepcell.org/
- Assay conditions & compute
- Inference was timed on one NVIDIA RTX A6000 GPU with eight AMD EPYC 7763 CPUs: about 40 s for a typical dataset of 10,000 detections over 30 frames, mostly segmentation. Training hardware and time are not reported.
- Reproducibility gaps
- Code: modified Apache license, non-commercial use only (the abstract says permissive); DynamicNuclearNet is non-commercial too. The authors' Fluo-N2DL-HeLa test labels are unreleased. Benchmark values are only in supplementary tables; no software versions.
Limitations (6)
- Segmentation remains the bottleneck: tracking is worse on predicted than on ground-truth masks, and the segmentation methods in wide use cannot handle overlapping objects. · stated by the authors
- Dataset diversity is limited compared with the full range of live-cell phenotypes; only nuclei are labeled, not whole cells. · stated by the authors
- Phenotype shifts under perturbation are expected to degrade segmentation and tracking, and the labeling framework does not yet capture phenotypes such as cell death. · stated by the authors
- Limited to 2D movies; extension to 3D live imaging is left to future work. · stated by the authors
- The benchmarks cannot separate how much of the performance comes from training-data size and how much from model architecture. · stated by the authors
- The comparison is uneven: Caliban was trained on DynamicNuclearNet, whose test split is the main benchmark, while comparators used existing models or parameters. The Fluo-N2DL-HeLa test labels are the authors' own, unreleased, and cover only two movies. · noted in this summary
Who should read:Read this if you need to segment and track fluorescent nuclei in 2D live-cell movies with division detection, or want a large labeled nuclear tracking dataset and division-specific tracking metrics.
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- LivecellX: Corrective Deep Learning for Object-Oriented Single-Cell Analysis in Live-Cell Imaging
Ke Ni, Gaohan Yu, … Jianhua Xing·bioRxiv·Feb 23, 2025
- Automated cell tracking using StarDist and TrackMate
E. Fazeli, N. H. Roy, … G. Jacquemet·bioRxiv·Sep 22, 2020
- Cell-ACDC: a user-friendly toolset embedding state-of-the-art neural networks for segmentation, tracking and cell cycle annotations of live-cell imaging data
F. Padovani, B. Mairhoermann, … K. M. Schmoller·bioRxiv·Sep 28, 2021
- Ultrack: pushing the limits of cell tracking across biological scales
Jordão Bragantini, Ilan Theodoro, … Löıc A. Royer·Nature MethodsPeer-reviewed·Aug 25, 2025
- DeepSea: An efficient deep learning model for single-cell segmentation and tracking of time-lapse microscopy images
Abolfazl Zargari, Gerrald A. Lodewijk, … S. Ali Shariati·Cell Reports MethodsPeer-reviewed·Jun 1, 2023
- Tracking cell lineages in 3D by incremental deep learning
Ko Sugawara, Çağrı Çevrim, Michalis Averof·eLifePeer-reviewed·Dec 8, 2021