Evidence-based re-identification: Leveraging appearance cues to correct tracking mistakes rather than trying to prevent them

Abstract

Naturalistic behavioral neuroscience experiments, in which groups of animals interact in enriched environments over prolonged periods, require each animal's identity to remain correct throughout the recording. Current multi-animal tracking tools assign identities frame by frame without revisiting past assignment decisions. A single, even sporadic, identity swap is therefore likely to persist until the end of the recording. Consequently, error-free tracking results become progressively less likely as recordings lengthen and cohorts grow. Even state-of-the-art multi-object tracking (MOT) methods that leverage appearance cues do so only to make each assignment more robust. This robustness, however, only delays the inevitable as even the most robust assignment process will eventually make a mistake over prolonged recordings. A mistake that will then persist just as it otherwise would have. Here, we introduce evidence-based re-identification, which uses appearance cues to correct tracking mistakes rather than prevent them. This fundamental shift allows the algorithm to build and progressively enrich appearance representations for each tracked subject. Tracking mistakes are then detected by continuously monitoring the evolution of these representations, triggering a correction whenever two representations drift toward one another in a sustained way. On two distinct, minutes-long benchmarks of mice interacting in enriched environments, motion-only trackers (DeepLabCut, SLEAP, ByteTrack) and the appearance-based BoT-SORT-ReID began accumulating identity errors within seconds, whereas ByteTrack paired with evidence-based re-identification remained reliable, outperforming even ByteTrack paired with the marker-based Tailtag system. This reliability was also positively correlated with the accuracy of the appearance extractor, but only when it was used through evidence-based re-identification. Foundation image encoders (CLIP, MegaDescriptor, DINOv3) showed promising out-of-the-box performance yet still required fine-tuning to be effective. Together, these results show that multi-animal tracking algorithms can remain reliable over prolonged periods only if they can reliably recover from tracking mistakes, and that fine-tuning appearance extractors remains essential to doing so.