Deep Imputation for Skeleton Data (DISK) for Behavioral Science

Abstract

AbstractPose estimation methods and motion capture systems have opened doors to quan- titative measurements of animal kinematics. However, these methods are not perfect and contain missing data. Our method, Deep Imputation for Skeleton data (DISK), leverages deep learning algorithms to learn dependencies between keypoints and their dynamics to impute missing tracking data. We developed an unsupervised training scheme, which does not rely on manual annotations, and tested several neural network architectures for the imputation task. We found that transformer outperforms other architectures including graph con- volutional networks that were developed specifically for skeleton-based action recognition. We demonstrate the usability and performance of our imputation method on seven different animal skeletons including two multi-animal set-ups. With an optional estimated imputation error, DISK enables behavior scien- tists to assess the reliability of the imputed data. The imputed recordings allow to detect more episodes of motion, such as steps, and to obtain more sta- tistically robust results when comparing these episodes between experimental conditions. While animal behavior experiments are expensive and complex, track- ing errors make sometimes large portions of the experimental data unusable. DISK allows for filling in the missing information and for taking full advantage of the rich behavioral data. This stand-alone imputation package, freely available at https://github.com/bozeklab/DISK.git, is applicable to results of any track- ing method (marker-based or markerless) and allows for any type of downstream analysis.