henhao
Sign in
Animal Behavior & CognitionMethods in Ecology and Evolution
Peer-reviewed journal article

YOLO-Behaviour: A simple, flexible framework to automatically quantify animal behaviours from videos

Alex Hoi Hang ChaniD, Prasetia Utama PutraiD, Harald Thomas SchuppiD, Johanna KöchlingiD, Jana StraßheimiD, Britta RenneriD, Julia SchroederiD, William D. PearseiD, Shinichi NakagawaiD, Terry A. BurkeiD, Michael GriesseriD, Andrea MeltzeriD, Saverio Lubrano, Fumihiro KanoiD

Abstract

Manually coding behaviours from videos is essential to study animal behaviour but it is labour-intensive and susceptible to inter-rater bias and reliability issues. Recent developments of computer vision tools enable the automatic quantification of behaviours, supplementing or even replacing manual annotations. However, widespread adoption of these methods is still limited, due to the lack of annotated training datasets and domain-specific knowledge required to optimize these models for animal research. Here, we present YOLO-Behaviour, a flexible framework for identifying visually distinct behaviours from video recordings. The framework is robust, easy to implement, and requires minimal manual annotations as training data. We demonstrate the flexibility of the framework with case studies for event-wise detection in house sparrow nestling provisioning, Siberian jay feeding, human eating behaviours, and frame-wise detections of various behaviours in pigeons, zebras, and giraffes. Our results show that the framework reliably detects behaviours accurately, and retrieve comparable accuracy metrics to manual annotation. However, metrics extracted for event-wise detection were less correlated with manual annotation, and potential reasons for the discrepancy between manual annotation and automatic detection are discussed. To mitigate this problem, the framework can be used as a hybrid approach of first detecting events using the pipeline and then manually confirming the detections, saving annotation time. We provide detailed documentation and guidelines on how to implement the YOLO-Behaviour framework, for researchers to readily train and deploy new models on their own study systems. We anticipate the framework can be another step towards lowering the barrier of entry for applying computer vision methods in animal behaviour.

PDFDOICodeData
Full-text digest ↓

02 /Specification & Classification

Click any tag to browse matching papers
Citations
32
FWCI
7.97
Field percentile
top 2%
OpenAlex Topics

In-depth summary

✦ Multi-agent verified full-text synthesis

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (27 Aug 2024) · CC BY 4.0

YOLOv8 detection with SORT event grouping and grid-search tuning quantifies visually distinct behaviours from video across 5 case studies, reaching f1-scores of 0.62 to 0.94.

Coding behaviour from video by hand is slow and suffers from poor observer reliability. Existing computer vision routes (keypoint pipelines, video classifiers, crop-then-classify tools) need large annotated datasets or multi-stage workflows; this work treats behaviour as single-frame object detection that needs only bounding-box labels, which makes it easier for biologists to adopt.

Key results

  1. ShownAfter tuning, the pipeline detected behaviours accurately on the event validation sets of all five case studies. These covered event-wise tasks (sparrow provisioning, jay feeding, human eating) and frame-wise tasks (pigeons, zebras and giraffes).

    Weighted-average f1-score ranged from 0.62 to 0.94 and accuracy from 0.70 to 0.98 across case studies (Table 3). Hyper-parameters were chosen by grid search to maximize f1-score. · Results; Tables

  2. ShownBehavioural metrics extracted automatically all correlated significantly with manual or dataset labels. Agreement was clearly higher for frame-wise than for event-wise case studies, and the automatic metrics generally under-counted behaviour.

    Pearson and ICC3 values were 0.78 to 0.92 for frame-wise sets (3D-POP pigeons, KABR zebras/giraffes) vs 0.49 to 0.70 for event-wise sets (sparrow visit rates, jay feeding rate, human eating rate). All Pearson p< 0.05. Figure 3 shows general under-detection. · Results; Evaluation; Tables

  3. ShownPer-class accuracy showed where the method struggles: human eating was likely confused with other hand-to-mouth gestures, and KABR movement behaviours (walking, trotting, running), browsing and auto-grooming were poorly resolved, probably due to missing motion context or highly variable postures.

    Per-class accuracy was 0.6 for human eating. In KABR it was 0.27-0.73 for walking/trotting/running, 0.069 for browsing and 0.37 for auto-grooming (confusion matrices). Pigeon walking and bowing were still detected reliably. · Results; Evaluation; Limitations

  4. ShownRe-tuning the hyper-parameters to minimize false negatives could let the pipeline serve as a first-pass screen before human review. It misses few events, though for human eating it yields many false positives that a person would then need to reject.

    False negative rates were 0.05 to 0.14 across event detection case studies (about 5% to 15% of events missed). For human eating: recall 0.86, false negative rate 0.14, precision 0.26. Manual confirmation could potentially raise precision to 0.86, vs 0.63 when fully automated. · Results; Hybrid applications; Tables

  5. SuggestedEven without temporal input, the single-frame framework reached accuracy on KABR not far below that reported for a temporal video model.

    The framework scored 72% accuracy on KABR (Table 3), vs 86.7% per-instance accuracy reported for the temporal X3D method in the original KABR publication; the X3D figure is cited, not rerun here. · Limitations

Methods, data and reproducibility

Behaviours are boxed on randomly sampled frames (for pigeon and KABR data, boxes and labels come from the existing datasets) and a YOLOv8-large detector is trained with Ultralytics defaults (70%, 20%, 10% split). Per-frame detections are merged into events with the SORT tracker. Confidence, minimum duration and SORT thresholds (min hits, max age, IOU) are tuned by grid search on an event validation set.

  • House sparrow nestling provisioning videos from Lundy Island, UK; male and female visit rates (visits per hour) · house sparrow (Passer domesticus) · 10.17617/3.EZNKYV
  • Siberian jay feeding videos from Swedish Lapland, manually coded in BORIS; feeding rate in seconds per minute per individual · Siberian jay (Perisoreus infaustus) · 10.17617/3.EZNKYV
  • Videos of humans eating, recorded in a specific experiment; eating rate (food eaten per minute); not released · human (Homo sapiens)
  • 3D-POP dataset of foraging homing pigeons recorded in Möggingen, Germany; existing frame-wise labels for five behaviours · homing pigeon (Columba livia)
  • KABR dataset of zebras and giraffes from Mpala research center, Kenya; existing frame-wise labels for seven behaviours · zebra (Equus quagga, E. grevyi); giraffe (Giraffa camelopardalis)
Methods & tools
YOLOv8-large object detector, Ultralytics Python package, SORT multi-object tracking algorithm, Grid-search hyper-parameter optimization, BORIS video annotation software (manual coding), X3D temporal video model (accuracy comparator from KABR publication), Pearson correlation, Intraclass correlation coefficient (ICC3)
Code
https://alexhang212.github.io/YOLO_Behaviour_Repo/
Data & accessions
https://doi.org/10.17617/3.EZNKYV
Reproducibility gaps
The human eating videos and their training annotations are withheld for privacy and ethical reasons. No software versions, hardware or runtimes are stated.
Limitations (6)
  • The YOLO model sees only one frame at a time, so it cannot reliably capture behaviours defined by motion, such as KABR walking, trotting and running. A speed threshold was proposed but could not be tested because KABR subjects were always centred in frame. · stated by the authors
  • The case studies used roughly 1000-1500 training images, but the framework may perform poorly for very rare behaviours with only a few instances. Few-shot object detection may suit such cases better. · stated by the authors
  • Event-wise metrics correlated less with manual coding than frame-wise ones, even where validation precision-recall was high (jays, sparrows). The authors cite BORIS timing mismatch (jays), many observers over the years (sparrows) and misidentified eating (humans); missed detections drove under-detection. · stated by the authors
  • Correlation and ICC values may not directly show how well behavioural metrics are predicted. The authors suggest further validation, such as hypothesis testing. · stated by the authors
  • Hyper-parameters and the 2s jay matching window were chosen on the same event validation set used to report detection accuracy. The Table 3 metrics are therefore likely optimistic compared with a truly held-out evaluation. · noted in this summary
  • There is no head-to-head comparison with other behaviour-quantification approaches (keypoint pipelines, LabGym, video classifiers) on the same data. The X3D comparison relies on a figure from a separate publication. · noted in this summary

Who should read:Read this if you want to automate video coding of visually distinct animal behaviours using a single-frame object detector and modest bounding-box annotation, or to combine automated screening with manual review.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISHarris '23Troscianko '25Shi '26Price '23Skovorodnik '25Silva '26
74% semantic matchNeuroscience
Carl Harris, Kelly R. Finn, Marie‐Luise Kieseler+2 more·Scientific Reports·Feb 15, 2023· 24 citations