henhaobio
Sign in

bioRxivAnimal Behavior & CognitionPreprint

DIPLOMAT: multi-animal tracking with efficient manual editing

Isaac RobinsoniD, George Glidden-HandgisiD, Neekesh Panchal, Nathan InseliD, Travis J. WheeleriD

PDFDOICode
In-depth full-text digest available · Jump to digest ↓

Abstract

Recent advances in computer vision have enabled the development of automated animal behavior observation tools. Several software packages currently exist for concurrently tracking pose in multiple animals; however, existing tools still face challenges in maintaining animal identities across frames and can demand extensive human oversight and editing. Here we report on DIPLOMAT, a Deep learning-based, Identity-Preserving, Labeled-Object Multi-Animal Tracker, which implements automated algorithms improving tolerance to occlusion and continuity of animal identity over a video, further supplemented by an efficient human interface to help eliminate remaining errors. DIPLOMAT is designed to perform multi-animal tracking by building on the per-frame pose prediction models of two state-of-the-art tools, DeepLabCut and SLEAP. Where other tools immediately take a maximum likelihood estimate from a given video frame, DIPLOMAT splits the probability fields according to the number of tracked animals and then applies an inference method that takes into account across-frame movement and probabilistic distances between body parts. These independent trace probabilities are then preserved for human editing, enabling multiple body parts to be re-tracked across frames with minimal user action. Testing with a standardized and independently tracked dataset of 3-mouse videos shows DIPLOMAT's automated components alone can reduce identity swaps by >75\%. DIPLOMAT code and documentation are available at https://diplomattrack.org/

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 3 (28 Aug 2026) · CC BY 4.0

A hidden Markov model trace over DeepLabCut or SLEAP confidence maps reduced identity swaps versus both tools on 54 MABe mouse-triplet videos; retained probabilities also guide manual editing.

Multi-animal pose trackers such as DeepLabCut (DLC), SLEAP and AlphaTracker assign identities frame by frame and stitch neighbouring frames, so similar-looking animals that touch or overlap get swapped or merged into chimeric skeletons, and editing them is laborious (in SLEAP, point by point for each frame). DIPLOMAT instead keeps the per-pixel probability maps, infers identities with a Viterbi trace over whole segments regardless of visual differences, and reuses the maps for editing and re-tracking.

Key results

  1. ShownAdding DIPLOMAT on top of the same DLC or SLEAP network lowered both single-body-part identity swaps and whole-body swaps with a 10-keypoint skeleton (BPM-10) on MABe mouse-triplet evaluation videos.

    54 evaluation videos of 60 s each, with ground truth from 5 independent human annotators per keypoint. Swaps were counted with the MotMetrics num switches metric and compared per video with paired Wilcoxon signed-rank tests (Tables 1 and 2). The means and p-values are in the tables, not in the prose. · Validation; Tables

  2. ShownWith a 5-keypoint skeleton (BPM-5) the gains were narrower. Body-part swaps fell relative to stand-alone DLC, and whole-body swaps fell relative to SLEAP, with only a strong statistical trend against DLC.

    Same 54 videos, Wilcoxon signed-rank tests (Table 1). The video set was able to separate the tools: at least one tracker made a full-body swap in 24 (BPM-10) and 23 (BPM-5) videos, and a body-part swap in 39 and 40 videos. · Validation; Tables

  3. ShownBeyond swaps, DIPLOMAT consistently raised identity recall for body parts and whole bodies and improved whole-body identity precision. Body-part precision improved over SLEAP but not over DLC. Distance to ground-truth points (MOTP) tended to be better than SLEAP's, with only minor differences from DLC.

    Means and Wilcoxon signed-rank p-values across the 54 MABe videos for individual body parts (Table 1) and whole bodies (Table 2). The prose gives no effect sizes. · Validation; Discussion; Tables

  4. SuggestedIn the cases where DIPLOMAT made more errors than the stand-alone tool, the mice had often stayed in physical contact for the whole 60 s video. That leaves no well-separated anchor frame from which to start the trace.

    The authors inspected the videos that fall below zero in the per-video improvement swarm plots (Figure 2A and 2C). They do not give a count of these videos. · Potential for errors; Figure legends

  5. ClaimedInteract sets user edits to 100% confidence and re-runs the Viterbi trace, so fixing a body swap often takes edits on only a few frames. The authors estimate the difference in manual editing time between SLEAP and DIPLOMAT at roughly 75% of video length.

    The editing-time estimate is given as roughly 75% of video length (±30% STD), with no described protocol, number of videos or users. The tool was also applied to rat and degu triads and dyads, with example degu frames from SLEAP v1.4.1 and DIPLOMAT Track v0.4.1 (Figure 4). · Interact: editing tracks using the DIPLOMAT…; Interact; Figure legends

Methods, data and reproducibility

Per-pixel body-part confidence maps from DLC or SLEAP bottom-up models serve as hidden Markov model emissions; a Viterbi trace per body part, within segments split at well-separated anchor frames, combines Gaussian movement transitions, skeletal-distance terms and a dominance rule keeping each animal's traces mutually independent; segments are stitched by the Hungarian algorithm, and a GUI re-traces after edits.

  • MABe 'Mouse triplet' training videos (60 s each), used to train the DLC and SLEAP models · mouse · n = Fifteen videos (27,000 frames)
  • MABe mouse-triplet evaluation videos (60 s each), keypoints annotated independently by 5 human workers, filtered to videos with positions for every body part in every frame · mouse · n = 54 videos
  • Rat recordings (triads and dyads), used for qualitative application of Track and Interact · rat
  • Degu recordings (triads and dyads), some with head implants or filmed in darkness with an IR camera; Figure 4 examples come from a 5 min video of 3 degus · degu
Methods & tools
DIPLOMAT (Track, Interact, Tweak), DeepLabCut (DLC), SLEAP (v1.4.1 for the Figure 4 examples), ONNX Runtime, MotMetrics, MABe Mouse triplet benchmark
Software & environment
Open source under the BSD 3-clause licence and installable from PyPI (diplomat-track). Inference runs through ONNX Runtime. Figure 4 used SLEAP v1.4.1 and DIPLOMAT Track v0.4.1. No pinned requirements or container are described.
Assay conditions & compute
Training and testing used two Xeon E5-2695v4 14-core processors, 192GB RAM and two Nvidia P100 GPUs with 16GB each. No runtimes are reported.
Reproducibility gaps
No URL or accession is given for the MABe videos. Trained models, evaluation scripts and per-video results are not stated as shared. No protocol is given for the editing-time estimate.
Limitations (6)
  • When animals stay in contact for a whole video there is no well-separated fix frame. The trace then starts without a meaningful prior, identities swap more easily, and DIPLOMAT can do worse than DLC or SLEAP alone. · stated by the authors
  • When animals overlap, probability fields can be split the wrong way and cause an identity swap. The authors doubt that any single-camera tool can track several animals without some identity confusion. · stated by the authors
  • In principle, if a body leaves a spot and quickly returns, the Viterbi trace could keep its parts in place instead of following the fast movement. The authors have not seen this with default parameters. · stated by the authors
  • The model currently has no notion of body-size variability. Its Gaussian movement transitions are static rather than fitted to instantaneous velocity, and it works only with single-camera 2D video. · stated by the authors
  • Quantitative validation uses one dataset (MABe mouse triplets, 54 videos). The degu application is shown only as qualitative example frames (Figure 4), and the rat application is only mentioned. · noted in this summary
  • The editing-time estimate (75% of video length, ±30% STD) comes without a described protocol, number of users or number of videos. Interact's efficiency gain is therefore not tested here. · noted in this summary

Read this if you track several similar-looking rodents from one camera with DeepLabCut or SLEAP and spend time fixing identity swaps when the animals touch or overlap.

Citations
2

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISLauer '22Torrents '25Walter '20Wang '25Azechi '24Pereira '20
79% semantic matchAnimal Behavior & Cognition
J. Lauer, M. Zhou, S. Ye+11 more·Nature Methods·Apr 12, 2022