Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network

Abstract

Quantitative ethology requires an accurate estimation of an organisms postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in novel behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation with much higher speed and comparable accuracy to the template-based approach, capturing detailed kinematics of 3-D larval swims. Author SummaryLarval zebrafish swimming has been studied extensively in 2-D environments, which are restrictive compared to natural 3-D habitats. To enable rapid capture of 3-D poses, we collect three orthogonal video projections of swim behaviors in several behavioral settings and fit poses to a physical model. We then use the physical model to generate an auto-annotated stream of synthetic poses to train a convolutional neural network. The network model performs highly accurate pose predictions on over 600 real swim bouts much faster than a physical model fit. Our results show that larvae frequently exhibit motions inaccessible in a 2-D setup. The annotated dataset could be used by ethologists studying larval swimming dynamics, and by the machine learning community interested in multi-dimensional time series and 3-D reconstruction. Using the ability to render images with multiple synthetic poses, our method can be extended to collective behavior.