bioRxivNeurosciencePreprint
Whole-body 3D kinematics of freely behaving Drosophila
Juan Ignacio IspizuaiD, Elliott T. T. AbeiD, Jinyao YaniD, Ratan OthayothiD, Steven C. Sawtelle, Felisha Atkins, Hiroshi ShiozakiiD, Niko R. MeieriD, Jovanka WongiD, Tiffany T. TraniD, Catherine K. MoriiD, Wilson Chen, Jakob VoigtsiD, David L. SterniD, Bingni W. BruntoniD, John C TuthilliD, Robert Evan JohnsoniD
Abstract
Understanding how nervous systems generate coordinated movement requires precise measurement of body kinematics during natural behavior. The fruit fly, Drosophila, is a model organism with sophisticated behavior and well-studied neural circuits, but tracking fly movements in 3D remains challenging because of their teeny bodies, rapid movements, and frequent self-occlusions. Here we present a pipeline for markerless, full-body 3D pose estimation of fly terrestrial behavior, combining seven synchronized high-speed cameras to capture whole-body kinematics at 800 frames per second. We trained a hybrid 2D/3D deep learning model to track 50 keypoints, then refined them to produce anatomically feasible kinematic trajectories through a retargeting process that solved an inverse kinematics problem constrained by a biomechanical body model. Analysis of 3D kinematics revealed that flies perform grounded running across their full speed range, without transitioning between discrete gaits. Using multi-animal tracking, we found that courting males coordinate both wings during song and modulate body pitch to track the female's vertical position. Our open-source pipeline and large 3D kinematic dataset of fly behavior provide a foundation for neuromechanical modeling and mechanistic studies of motor control in a genetically tractable model organism.
- organism: Drosophila
- assay: imaging
- research focus: software / tool
- research focus: database / resource
- research focus: cell type annotation
- research focus: trajectory inference
- approach: deep learning
- approach: transformer / foundation model
- approach: experimental assay
In-depth summary
Claude Opus 5.5 · xhigh effort · 7 Nov 2026 · from the full text of version 2 (16 Sept 2026) · CC BY 4.0
Seven high-speed cameras, a hybrid 2D/3D network and body-model inverse kinematics track 50 keypoints on freely moving flies; height and speed peak together at all running speeds.
Before this work, high-resolution 3D joint tracking of flies required tethering, which changes behavior and locomotor biomechanics, while untethered flies were tracked only in 2D or for selected parts such as leg tips. This pipeline records whole-body 3D kinematics of untethered flies, which the authors argue MuJoCo-based neuromechanical fly models need for imitation learning and validation.
Key results
ShownOn held-out running frames, triangulated foreleg-tip predictions reproject with a mean error of 18.2 µm across seven camera views, about 2-3% of the length of a fly femur.
Front tarsal tips were hand-labelled in all 7 views in 51 frames: reprojection error 1.48 ± 0.69 pixels (18.2 ± 8.5 µm, mean ± s.d.), against a femur of ~0.72 mm. Calibration landmarks: 0.48 ± 0.26 pixels (5.9 ± 3.2 µm) for N=272 landmarks, computed on the calibration training data. Cameras resolve ~12.3 µm/pixel at 800 fps. · 2.1 The rainbow rig tracks whole-body 3D…; Camera calibration
SuggestedScutellum height and forward speed (used as a stand-in for the centre of mass) peak in phase twice per step cycle at every running speed, with no aerial phase. This fits grounded running: one locomotor program adjusted continuously with speed, with no switching between discrete gaits.
2213 running bouts (~654,000 frame-sets) from 53 flies. Peaks fall at 0.5-0.6 rad (mid L1 swing) and -2.8 to -2.9 rad (mid L1 stance), each during one tripod's swing. Mean phase and spread change little across speed tertiles. Oscillation amplitude rises with step-cycle mean speed (linear-fit slopes 0.159 for height, 0.567 for speed) while the phase relationship stays stable. · 2.2 3D joint kinematics of freely running flies; 2.3 Fly kinematics are consistent with grounded…; Amplitude vs. speed; Figure legends
ShownCoupling between legs tightens with speed. Tripod groups share phase offsets in fast running, while slow running shows weaker, more variable coupling, most clearly in the hind legs. The increase in joint range of motion with speed falls on different joints in the middle legs than in the front and hind legs.
Median mean resultant length R of each leg's phase offset to L1 rises from the slow to the fast tertile for every leg (R1 0.70 to 0.93, L2 0.72 to 0.95, R2 0.73 to 0.95, L3 0.69 to 0.93, R3 0.63 to 0.93; 33-35 paired flies per leg). Two-sample Kuiper tests: V = 0.30-0.57, all p < 0.01 after Bonferroni correction. · 2.2 3D joint kinematics of freely running flies; Inter-leg coordination across speeds; Joint-speed correlation analysis; Figure legends
ShownIn courting pairs tracked from video alone, the unextended wing moves in phase with the extended, singing wing during sine song, and the phase does not break when the singing wing switches. Male thorax pitch follows the angle from the male's body to the female's centre of mass.
11 male-female pairs filmed at 800 Hz. During sine song, phase differences between the wings cluster near 0 rad, and amplitude passes from one wing to the other at switches with no phase discontinuity. Pulses detected from wing-tip kinematics split into Pslow and Pfast under a two-component Gaussian mixture. Pitch versus angle to the female's centre of mass: median offset 23.2° across bouts. · 2.4 Multi-animal tracking of flies during…; 3.2 Courtship kinematics in three dimensions; Courtship data; Figure legends
SuggestedWith little extra annotation, the pipeline also tracks grooming, optogenetically driven running in headless flies, and running over stepped terrain. Running induced through DNg100 uses less leg flexion than wild-type running but stays inside the wild-type kinematic space.
About 600 extra annotated frames were enough for grooming. Midleg stance centroids shift 0.41 mm (L2) and 0.47 mm (R2), p = 0.002 (Hotelling T², Holm-Bonferroni; 14 grooming vs 19 standing bouts, one male fly). DNg100>CsChrimson vs speed-matched wild type: principal-component (PC) centroids 0.17 PC units apart, n=50 cycles each. On steps, flies searched with the forelegs and swung legs again after failed landings. · 2.5 3D tracking of other behaviors and…; Figure legends
Methods, data and reproducibility
Seven synchronized high-speed cameras with telecentric lenses and strobed LEDs film flies at 800 fps in a glass-walled, 3D-printed chamber. A JARVIS hybrid 2D/3D CNN predicts 50 keypoints, and STAC-mjx inverse kinematics fits them to a MuJoCo fly body model to produce joint-angle trajectories. Courting pairs go through a multi-animal pipeline (CenterDetect, a DINOv3-based multi-view transformer, SAM3).
- Multi-camera 800 fps video and 3D keypoint tracks of freely behaving wild-type Canton-S adults (some F1 of Canton-S x empty split-Gal4) during running, grooming and courtship · Drosophila melanogaster · n = 43 male and 32 female adult flies, 2-5 days old
- Curated running dataset (bouts with at least two complete step cycles), analysed as STAC-refined joint angles · Drosophila melanogaster · n = 2213 running bouts (~654,000 frame-sets) from 53 flies (32 males, 21 females)
- Courtship recordings of male-female pairs (males 4-5 days old, females 1-2 days old) · Drosophila melanogaster · n = 11 pairs
- JARVIS training annotations for the 50-keypoint skeleton, labelled in red with triangulation across views · Drosophila melanogaster · n = ~2,000 frames/camera (~14,000 images, ~4,000 by hand); ~600 more for grooming
- Grooming versus standing bouts used for the head-angle and midleg-shift analyses · Drosophila melanogaster · n = 14 grooming (16,072 frames) and 19 standing bouts (2,123 frames); one male fly
- Optogenetically induced running in decapitated DNg100 split-Gal4 > UAS-CsChrimson flies, compared with speed-matched wild-type running · Drosophila melanogaster · n = 50 running cycles per condition
- Running over 3D-printed stepped terrain inside the chamber · Drosophila melanogaster
- 3D-printed calibration object with 23 landmarks, imaged through the glass arena walls · n = N=272 landmarks
- Methods & tools
- JARVIS HybridNet, red (multi-camera annotation software), orange (multi-camera acquisition software), STAC-mjx, MuJoCo fly body model, DLTdv (adapted for telecentric lenses), CenterDetect (EfficientNet), DINOv3 ViT-B/16, SAM3, FlySongSegmenter, CsChrimson optogenetics, Saturn IV Light Array
- Data & accessions
- https://bit.ly/3UWoXBF
- Experimental conditions
- Acquisition ran on one Threadripper PRO workstation with two NVIDIA A16 GPUs, compressing up to eight 25GigE camera streams to H.264 in real time. Compute for model training and inverse kinematics is not stated.
- Reproducibility gaps
- The text says hardware designs, calibration procedures and trained JARVIS models are released openly, but gives no links beyond the two GitHub locations. The dataset is reached through a URL shortener, with no DOI or accession. Software versions are not listed.
Limitations (6)
- The 24 × 5.5 × 9.5 mm corridor limits recordings to short locomotor sequences. Longer or more naturalistic tracking would need larger arenas with more cameras, or a system that follows the animal. · stated by the authors
- The 50-keypoint skeleton does not resolve individual tarsal or antennal segments. Tracking them would need higher magnification, which would shrink an already small field of view. · stated by the authors
- Only kinematics are measured, not ground reaction or muscle forces, so biomechanical models trained on these data still lack ground-truth dynamics. · stated by the authors
- The white strobe light used for imaging also activates CsChrimson, so illumination and optogenetic stimulation cannot be set independently. · stated by the authors
- Accuracy on held-out frames was checked only for front tarsal tips (51 frames from one running bout). No error is reported for wing, head or abdomen keypoints, or for the multi-animal courtship model. · noted in this summary
- Several conclusions rest on proxies or small samples. The scutellum stands in for the centre of mass in the grounded-running analysis, the grooming leg shift comes from one male fly, and the DNg100 comparison uses 50 cycles per condition. · noted in this summary
Read this if you study insect locomotion or courtship motor control, build neuromechanical models of Drosophila, or need multi-camera 3D pose estimation for small, fast-moving animals.
- Citations
- 0
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Precise kinematic and muscle recording in freely behaving flies enabled by closed-loop tracking and annotation-free pose estimation
Sibo Wang-Chen, Victor Alfred Stimpfling, … Pavan P Ramdya·bioRxiv·Mar 14, 2026
- FLiTrak3D: Improved deep-learning-based 3D insect flightkinematics tracking using spatial and temporal encoding
Antoine Cribellier, Abel-John Buchner·bioRxiv·Sep 10, 2026
- DeepFly3D: A deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila
- FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila
Virginie Uhlmann, Pavan P Ramdya, … Michael A. Unser·PLoS ONEPeer-reviewed·Apr 28, 2017
- A leg model based on anatomical landmarks to study 3D joint kinematics of walking in Drosophila melanogaster
M. Haustein, A. Blanke, … A. Buschges·bioRxiv·Apr 4, 2024
- Pose Estimation of Free-Flying Fruit Flies
T. Beatus, O. Ben-Dov·bioRxiv·Jan 24, 2021