TracktorLive: an integrated real-time object tracking and response system

Abstract

Real-time tracking and automated response systems are essential for standardising experiments, reducing observer bias, and improving reproducibility in studies of movement and behaviour. However, existing solutions face significant challenges: AI-based tracking systems require expensive hardware and impose computational delays, creating challenges for closed-loop experiments; existing real-time tracking tools lack standardised implementations for response delivery; and steep learning curves limit accessibility for users without programming or computer vision expertise. Here, we introduce TracktorLive, an open-source Python package designed to overcome these challenges through concurrency and a modular, cassette-based architecture. TracktorLive leverages traditional computer vision techniques to perform image-based object detection without the need for expensive hardware or deep learning. By parallelizing object tracking and response delivery into separate, concurrent server and client processes, the software minimizes frame processing time, enabling rapid, real-time analysis and response delivery. User-friendly cassettes--portable code snippets that can be copy-pasted into scripts--enable users with minimal programming experience to implement complex workflows for use in experiments and practical applications. We demonstrate TracktorLives utility through several applications, including microcontroller-based stimulus delivery for location-dependent manipulations; conditional video recording that activates only during events of interest; kinematic-based response triggering using real-time velocity computations; and multi-cassette experimental designs combining multiple functionalities. Detailed tutorials are provided to familiarize users with TracktorLives operation and functionality, and a growing library of cassettes supports diverse applications for processing both real-time and pre-recorded video. We validated the software by comparing its response timing to human experimenters in a stimulus delivery task involving two fish species, where TracktorLive demonstrated consistently higher accuracy and lower variability, particularly for fast-moving subjects. Beyond experimental biology, TracktorLives unique architecture and versatility could support many different applications in fields ranging from neuroscience to wildlife management. As an open-source software combining accessibility, modularity, and computational efficiency, TracktorLive can help democratize real-time tracking and automated response systems across disciplines.