PoseR - A deep learning toolbox for decoding animal behavior

Abstract

The actions of animals provide a window into how their minds work. Recent advances in deep learning are providing powerful approaches to recognize patterns of animal movement from video recordings using markerless pose estimation models. There is an increasingly rich field of unsupervised and supervised methods for classifying animal behavior built upon the outputs of pose estimation models. However, these methods often rely on species and task-specific feature engineering of trajectories, kinematics and task programming. Highly generalized solutions that use only pose estimations and the inherent structure of animals and their environment provide an opportunity to develop foundational, contextual and, importantly, standardized animal behavior models for efficient and reproducible behavioral analysis. Here, we present PoseRecognition (PoseR), a behavioral classifier leveraging action recognition models using spatio-temporal graph convolutional networks. We show that it can be used to classify animal behavior quickly and accurately from pose estimations, using zebrafish larvae, Drosophila melanogaster, mice, and rats as model organisms. PoseR can be accessed using a Napari plugin, which facilitates efficient behavioral extraction for bout-like behaviour, annotation, model training and deployment. Our tool simplifies the behavioral analysis workflow by transforming coordinates of animal position and pose into semantic labels with speed and precision. Furthermore, we contribute a novel method for unsupervised clustering of behaviors and provide open-source access to our zebrafish datasets and models. The design of our tool ensures scalability and versatility for use across multiple species and contexts, improving the efficiency of behavioral analysis across fields.