henhaobio
Sign in

PLoS ONEAnimal Behavior & CognitionPeer-reviewed journal article

An open-source closed-loop Virtual Reality system to investigate social interactions and collective behavior in fish

Stéphane SancheziD, Ramón EscobedoiD, Renaud BastieniD, Boris A. J. Lenseigne, Audrey Denis, Mathieu MoreauiD, Maud CombeiD, Andrew David StrawiD, Clément SireiD, Guy Theraulaz AiD

PDFDOIPreprintCode

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (10 Dec 2025) · CC BY-NC-ND 4.0

Real rummy-nose tetras follow a projected, anamorphically rendered virtual conspecific in a low-cost open-source 3D-tracked VR bowl, largely matching its speed and depth; a moving sphere elicits weak following.

Probing how a fish reacts to a neighbour needs a social partner whose behaviour is controlled, and the paper argues that observation and open-loop setups miss the two-way influence between animal and stimulus. Relative to the earlier FishVR virtual reality design, this rig uses a bigger bowl suited to adult rummy-nose tetras and a single depth camera, with hardware plans and tracking, simulation, rendering and calibration code released openly.

Key results

  1. SuggestedFollowing is specific to the fish image: real fish stayed nearer to, and moved more in step with, a virtual conspecific than a similar-sized dark sphere moving on the same circle.

    Mean real-to-virtual distance 9.3 cm ± 7.0 SD with the fish model vs 15.3 cm ± 7.2 SD with the sphere; peak velocity cross-correlation C ≈ 0.77 vs C ≈ 0.27, at lags of −0.133 s and −0.067 s. Path radius 10 cm, 10 cm/s, depth 5 cm; 7 one-hour repetitions, fish model for 30 min then sphere for 30 min. · 3.6.2 Experimental procedure; 4.1 Control condition with a non-social stimulus; Figure legends

  2. ShownReal fish adjusted their speed to the virtual fish. At 10 cm/s (which the authors call the comfort speed) they followed closely with negligible lag. A 5 cm/s partner was overshot and circled, and at 15 cm/s they lagged or cut corners. Separation and depth stayed the same throughout.

    Real-fish mean speeds 6.3, 9.4 and 11.7 cm/s for virtual speeds 5, 10 and 15 cm/s; lag at peak cross-correlation 0.4 s, 0 s and −0.267 s respectively. Inter-fish distance probability density function (PDF) peaks near 2.5 cm and depth near 4.5 cm in all three; speed PDFs differ (Hellinger distance > 0.2), distance and depth PDFs do not. 10 one-hour repetitions per condition. · 4.2.1 Impact of the virtual fish’s swimming…; 8.1 Captions of the Supplementary Figures; 8.2 Captions of the Supplementary Tables; Figure legends

  3. ShownReal fish tracked the virtual fish's depth, usually swimming slightly above it. With the partner near the bottom, the smaller vertical room shrank the area the real fish covered and reduced mean separation, while the most frequent separation stayed the same.

    Virtual depths 3.5, 5 and 8 cm at 10 cm/s: mean inter-fish distance 7.97, 7.83 and 6.25 cm; distance PDF peak near 2.5 cm in all; x-range covered around the virtual fish (radius of the density maps) reaches about 38 cm in the shallower cases vs 15 cm at 8 cm depth; depth deviations rarely beyond ±1.5 cm. Hellinger distance > 0.2 for inter-fish distance at 8 cm and for real-fish depth across conditions. · 4.2.2 Impact of virtual fish’s swimming depth on…; 8.2 Captions of the Supplementary Tables; Figure legends

  4. SuggestedWith the virtual fish 10.4 cm from the wall, the real fish often swam between it and the wall. It was responding to an image projected on the far side of the bowl and seen through the water, which the authors read as evidence of a convincing anamorphic illusion.

    Trajectories and relative-position maps (Figs 7D, 8, 11) with wall distances from 2.4 to 10.4 cm at 10 cm/s and 5 cm depth; real-fish speed and depth distributions did not differ across wall distances, and only the 2.4 cm condition (C7) gave a larger inter-fish distance than 10.4 cm (C6), with Hellinger distance > 0.2. · 4.2 Behavioral response of real fish to the…; 4.2.3 Impact of the virtual fish’s distance to…; 8.2 Captions of the Supplementary Tables; Figure legends

  5. SuggestedReal fish also followed virtual fish on three-petal rose (rhodonea) paths that cross the optically distorted bowl centre. They followed less well when the petals were sharp, and tended to sit on the inner side of the partner's turn.

    Peak velocity cross-correlation C ≈ 0.65 for broad-petal Rose 1 (n = 3, d = 5) vs C ≈ 0.51 for sharp-petal Rose 2 (n = 3, d = 1), at lags −0.23 s and −0.1 s; inter-fish distance PDF still peaks at 2.5 cm; speed PDFs centred on 10 cm/s but wider than for circles; perception-angle PDFs asymmetric toward the turn side. 5 one-hour repetitions per rose. · 4.3 Behavioral response of real fish to the…; 8.1 Captions of the Supplementary Figures; Figure legends

Methods, data and reproducibility

One rummy-nose tetra swims freely in a 15 L spherical acrylic bowl, tracked in 3D by an Intel RealSense D435; a C++ Qt simulator drives a virtual fish on fixed circle or rose paths and a Unity renderer projects it from below, anamorphically corrected for the real fish's position. Responses are analysed with Hellinger distances between probability density functions (PDFs) and velocity cross-correlation.

  • Control trials: virtual conspecific for 30 min, then a dark virtual sphere for 30 min on the same circular path · Hemigrammus rhodostomus (rummy-nose tetra) · n = 7 one-hour repetitions
  • Circular-trajectory trials, 7 conditions varying virtual fish speed, depth or distance to the wall · Hemigrammus rhodostomus (rummy-nose tetra) · n = 10 one-hour repetitions per condition
  • Rhodonea (rose curve) trajectory trials, 2 conditions · Hemigrammus rhodostomus (rummy-nose tetra) · n = 5 one-hour repetitions per condition
  • Tracking data from all experiments, deposited on figshare · Hemigrammus rhodostomus (rummy-nose tetra) · 10.6084/m9.figshare.29097605
Methods & tools
Intel RealSense D435, RealSense SDK, OpenCV, MOG2 background subtraction, Random Matrix Tracker, Kalman filter, Python, Qt (C++), Unity (C#), OPTOMA WUXGA DLP LED projector, Hellinger distance
Assay conditions & compute
One Dell Precision 3640 Tower workstation with an NVIDIA RTX 3070 graphics card handles tracking, trajectory simulation, environment computation and display in real time; tracking runs at 30 to 90 frames per second.
Reproducibility gaps
Experimental data are released on figshare, but the text prints only a bare DOI (10.6084/m9.figshare.29097605, listed under methods.data), not a URL. No software or library versions, dependency files or containers are stated, and the number of distinct fish tested is not reported.
Limitations (6)
  • Only one real fish and one virtual fish were tested; extending the system to several virtual fish or other species is described as straightforward but is not demonstrated here. · stated by the authors
  • Virtual fish followed preset (open-loop) circle and rose paths, and only the anamorphic rendering tracked the real fish; control of the virtual fish by a behavioural model is supported but left for future work. · stated by the authors
  • The D435's onboard depth map appears to round some disparities to integers, creating artificial depth bands; the authors' own stereo computation from the IR frames removes this but is noisier and needs Kalman smoothing. · stated by the authors
  • Bowl geometry constrains behaviour: a deep virtual fish leaves little vertical room, and paths through the bowl centre are seen through more water, adding optical distortion. · stated by the authors
  • The sphere control always came after 30 min with the virtual fish in the same session, so order or habituation effects are not separated from stimulus type; this comparison rests on 7 repetitions. · noted in this summary
  • Conditions are compared with pooled distributions and a fixed Hellinger threshold (H > 0.2) rather than tests across individuals; the number of distinct fish is not given, and each rose condition has 5 repetitions. · noted in this summary

Read this if you want to run controlled social-stimulus experiments on freely swimming adult fish and need an open, documented hardware and software stack for 3D tracking and anamorphic projection.

Citations
6
FWCI
23.57
Field percentile
top 1%

Abstract

1This study introduces a low-cost, open-source, immersive Virtual Reality (VR) system designed to investigate real-time social interactions and collective behaviors in fish. Understanding collective animal behaviors, such as schooling in fish, presents significant observational challenges due to rapid and complex interactions. To address these difficulties, we developed an innovative closed-loop VR environment allowing precise control and measurement of interactions between real and virtual fish. This setup incorporates high-speed 3D tracking, real-time visual feedback, and automated data processing, creating realistic and interactive scenarios. We present experimental results, obtained with rummy-nose tetras (Hemigrammus rhodostomus ), showing that freely moving real fish consistently adjusted their speed, depth, and spatial positioning to follow virtual fish effectively. Fish matched moderate virtual speeds comfortably, struggled slightly at higher speeds, and actively maintained vertical positioning to sustain group cohesion. The implementation of this VR system provides critical insights into sensory and cognitive processes underlying collective movement and decision-making in fish. By revealing both flexibility and constraints in fish behavior, our findings underscore the potential of VR technology to study social interactions in controlled yet realistic settings. Ultimately, this VR system advances our understanding of collective animal behavior, providing a foundation for cross-species studies linking neural processes, behavioral interactions, and group dynamics.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Similarity Map
SelectedSame fieldCross-field
THISEscobedo '26Escobedo '26Vidal '22Kang '26Javier '26Papaspyros '26
77% semantic matchAnimal Behavior & Cognition
Ramón Escobedo, Justine Reynaud, Stéphane Sanchez+2 more·Proceedings of the Royal Society B Biological Sciences·Jun 3, 2026· 2 citations