henhaobio
Sign in

Smart Agricultural TechnologyAnimal Behavior & CognitionPeer-reviewed journal article

Impact of cross-validation designs on cattle behavior prediction using machine learning and deep learning models with tri-axial accelerometer data

Jin WangiD, Ziwen YuiD, Ricardo Carbonari ChebeliD, Haipeng YuiD

PDFDOIPreprint

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (23 Jan 2025) · CC BY 4.0

Leave-cow-out validation lowered cattle behavior classification accuracy to 0.72 to 0.82, versus 0.9 to 0.96 under random hold-out splits that can place one animal's records in both sets.

Cattle behavior classifiers built from accelerometer data are usually evaluated with random cross-validation (CV), which can put records of one animal in both training and test sets and overstate accuracy; animal-blocked CV had been little studied for imbalanced multi-class behavior. It also compares classical machine learning (ML) and deep learning (DL) models under block CV, which, per the authors, earlier studies had only done with random splits.

Key results

  1. ShownEvery model scored lower when whole cows were withheld for testing (leave-cow-out, a block design) than under random hold-out splits. Random partitioning therefore gave an optimistic estimate of behavior classification performance.

    Hold-out: ML accuracy 0.94 to 0.96, macro F1 0.93 to 0.95; DL accuracy 0.9 to 0.92, F1 0.89 to 0.91. Leave-cow-out: ML accuracy 0.72 to 0.82, F1 0.64 to 0.82; DL accuracy 0.76 to 0.82, F1 0.64 to 0.76. Six beef cows, four behaviors. · Abstract; 4 Results; 5.1 Comparison of prediction performance between…; 6 Conclusion

  2. SuggestedWith unseen cows in the test set, no single model led throughout. The authors report that the multi-layer perceptron (MLP) matched or beat the ML models, most clearly with three cows withheld; with one cow withheld, RF led and the convolutional neural network (CNN) came second.

    Leaving one (L1CO), two (L2CO) or three (L3CO) cows out. L1CO: random forest (RF) accuracy 0.82, F1 0.7; CNN 0.81, 0.66. L2CO: RF accuracy 0.79, MLP and CNN 0.78; best F1 gradient boosting (GB) 0.82, RF 0.73. L3CO: MLP accuracy 0.82, F1 0.76 vs RF 0.77, 0.71. Accuracy tended to drop with more withheld cows; F1 showed no clear trend. · 4 Results; 5.1 Comparison of prediction performance between…; 6 Conclusion

  3. ShownUnder random hold-out splits, the three classical models (k-nearest neighbors (KNN), RF, GB) slightly but consistently outperformed both DL models. Scores fell only modestly as the training share shrank from 90% to 50%.

    90% training: GB accuracy 0.96, F1 0.95; KNN 0.95 and 0.95; RF 0.95 and 0.94; MLP and CNN each accuracy 0.91, F1 0.9. Training-to-testing ratios tested: 9:1, 8:2, 7:3, 6:4 and 5:5. · 3.4.1 Cross-validation designs; 4 Results

  4. SuggestedThe authors attribute the hold-out advantage to dependence between records from the same cow placed in both training and test sets. They recommend block CV when predicting for unseen animals or new farms.

    Inferred from the consistent gap between hold-out and leave-cow-out results rather than from a dedicated experiment. The authors note agreement with Roberts et al., Wang and Bovenhuis (farms as blocks) and Coelho Ribeiro et al., who also found random CV more optimistic than block CV. · 5.2 Impact of CV designs on prediction…; 6 Conclusion

Methods, data and reproducibility

Tri-axial acceleration from a neck-mounted sensor was expanded to seven instantaneous signals, summarized in 1 s sliding windows (0.5 s step) into 20 low-collinearity features, and classified into four behaviors by three classical and two neural models. Models were compared under random hold-out and leave-one/two/three-cow-out splits, with hyperparameters tuned by 5-fold cross-validation.

  • Public neck-mounted tri-axial accelerometer data (25 Hz) with video-annotated labels; 13 annotated behaviors reduced to four (standing rest, standing rumination, moving, grazing) · cattle (beef cows) · n = six cows; 22, 987 records over 197 min
Methods & tools
k-nearest neighbors (scikit-learn), random forest (scikit-learn), gradient boosting classifier (scikit-learn), multi-layer perceptron (TensorFlow), convolutional neural network (TensorFlow), Adam optimizer with categorical cross-entropy loss, grid search with 5-fold cross-validation, Kionix KX122-1037 tri-axial accelerometer
Reagents & environment
Python with scikit-learn (KNN, RF, GB) and TensorFlow (MLP, CNN); package versions are not stated.
Reproducibility gaps
No code repository or analysis scripts are referenced; the dataset is described as public on Zenodo but no record identifier or URL is printed; software versions are not given.
Limitations (4)
  • The authors caution that the MLP's apparent robustness on unseen cows is preliminary and must be confirmed on a larger, more comprehensive dataset. · stated by the authors
  • The authors note that no earlier study compared ML and CNN models under leave-cow-out CV for cattle behavior, so that part of their comparison has no direct precedent to check against. · stated by the authors
  • Only six cows and 197 min of data are used, each leave-cow-out fold tests one to three animals, and the text reports no fold-level variance, confidence intervals or significance tests, so small ranking gaps (e.g., 0.82 vs 0.81 accuracy) may not be robust. · noted in this summary
  • The 1 s windows advance in 0.5 s steps, so neighboring windows share raw samples and can land in both hold-out sets. No within-cow temporal-block design was tested, so the gap mixes window-overlap leakage with between-cow generalization. · noted in this summary

Read this if you validate accelerometer or other wearable-sensor classifiers of livestock behavior and need evidence on how animal-blocked versus random splits change the accuracy you report.

Citations
7
FWCI
3.81
Field percentile
top 6%

Abstract

1Assessing cattle behaviors provides insights into animal health, welfare, and productivity to support on-farm management decisions. Wearable accelerometers offer an alternative approach to traditional human evaluation, providing a more objective and efficient method for predicting cattle behavior. Random cross-validation (CV) is commonly used to evaluate behavior prediction by splitting data into training and testing sets, but it can yield inflated results when records from the same animal are included in both sets. Block CV splits data by block effects, offering a more realistic evaluation but remains underexplored for predicting multi-class imbalanced cattle behavior. Additionally, deep learning (DL) models have not been fully explored for behavior prediction compared to machine learning (ML) models. The objectives of this study were to examine the impact of CV designs on multi-class imbalanced cattle behavior prediction and to compare the performance of ML and DL models. Three ML and two DL models were used to predict the four behaviors of six beef cows from a public tri-axial accelerometer dataset, with model performance evaluated using both hold-out and leave-cow-out CV designs representing random and block CV, respectively. In hold-out CV, ML models achieved accuracies of 0.94 to 0.96 and F1 scores of 0.93 to 0.95, while DL models achieved accuracies of 0.9 to 0.92 and F1 scores of 0.89 to 0.91. In the leave-cow-out CV, ML models obtained accuracies of 0.72 to 0.82 and F1 scores of 0.64 to 0.82, whereas DL models obtained accuracies of 0.76 to 0.82 and F1 scores of 0.64 to 0.76. Generally, ML models outperformed DL models in the hold-out CV, but the multi-layer perceptron DL model demonstrated comparable or superior performance in the leave-cow-out CV. All models performed better with hold-out CV than leave-cow-out CV. Our results suggest that CV designs can affect behavior prediction performance. While a random CV produces seemingly good predictions, these results can be artificially inflated by the data partition. A block CV that strategically partitions data could be a more appropriate design.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISAgarwal '25Neupane '24Bloch '22Chambers '20Nagy '23Rast '19
69% semantic matchEcology
Medha Agarwal, Kasim Rafiq, Ronak Mehta+2 more·Methods in Ecology and Evolution·Dec 8, 2025· 3 citations