PLoS ONEAnimal Behavior & CognitionPeer-reviewed journal article
An open-source closed-loop Virtual Reality system to investigate social interactions and collective behavior in fish
Stéphane SancheziD, Ramón EscobedoiD, Renaud BastieniD, Boris A. J. Lenseigne, Audrey Denis, Mathieu MoreauiD, Maud CombeiD, Andrew David StrawiD, Clément SireiD, Guy Theraulaz AiD
In-depth summary
Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 2 (10 Dec 2025) · CC BY-NC-ND 4.0
Real rummy-nose tetras follow a projected, anamorphically rendered virtual conspecific in a low-cost open-source 3D-tracked VR bowl, largely matching its speed and depth; a moving sphere elicits weak following.
Probing how a fish reacts to a neighbour needs a social partner whose behaviour is controlled, and the paper argues that observation and open-loop setups miss the two-way influence between animal and stimulus. Relative to the earlier FishVR virtual reality design, this rig uses a bigger bowl suited to adult rummy-nose tetras and a single depth camera, with hardware plans and tracking, simulation, rendering and calibration code released openly.
Key results
SuggestedFollowing is specific to the fish image: real fish stayed nearer to, and moved more in step with, a virtual conspecific than a similar-sized dark sphere moving on the same circle.
Mean real-to-virtual distance 9.3 cm ± 7.0 SD with the fish model vs 15.3 cm ± 7.2 SD with the sphere; peak velocity cross-correlation C ≈ 0.77 vs C ≈ 0.27, at lags of −0.133 s and −0.067 s. Path radius 10 cm, 10 cm/s, depth 5 cm; 7 one-hour repetitions, fish model for 30 min then sphere for 30 min. · 3.6.2 Experimental procedure; 4.1 Control condition with a non-social stimulus; Figure legends
ShownReal fish adjusted their speed to the virtual fish. At 10 cm/s (which the authors call the comfort speed) they followed closely with negligible lag. A 5 cm/s partner was overshot and circled, and at 15 cm/s they lagged or cut corners. Separation and depth stayed the same throughout.
Real-fish mean speeds 6.3, 9.4 and 11.7 cm/s for virtual speeds 5, 10 and 15 cm/s; lag at peak cross-correlation 0.4 s, 0 s and −0.267 s respectively. Inter-fish distance probability density function (PDF) peaks near 2.5 cm and depth near 4.5 cm in all three; speed PDFs differ (Hellinger distance > 0.2), distance and depth PDFs do not. 10 one-hour repetitions per condition. · 4.2.1 Impact of the virtual fish’s swimming…; 8.1 Captions of the Supplementary Figures; 8.2 Captions of the Supplementary Tables; Figure legends
ShownReal fish tracked the virtual fish's depth, usually swimming slightly above it. With the partner near the bottom, the smaller vertical room shrank the area the real fish covered and reduced mean separation, while the most frequent separation stayed the same.
Virtual depths 3.5, 5 and 8 cm at 10 cm/s: mean inter-fish distance 7.97, 7.83 and 6.25 cm; distance PDF peak near 2.5 cm in all; x-range covered around the virtual fish (radius of the density maps) reaches about 38 cm in the shallower cases vs 15 cm at 8 cm depth; depth deviations rarely beyond ±1.5 cm. Hellinger distance > 0.2 for inter-fish distance at 8 cm and for real-fish depth across conditions. · 4.2.2 Impact of virtual fish’s swimming depth on…; 8.2 Captions of the Supplementary Tables; Figure legends
SuggestedWith the virtual fish 10.4 cm from the wall, the real fish often swam between it and the wall. It was responding to an image projected on the far side of the bowl and seen through the water, which the authors read as evidence of a convincing anamorphic illusion.
Trajectories and relative-position maps (Figs 7D, 8, 11) with wall distances from 2.4 to 10.4 cm at 10 cm/s and 5 cm depth; real-fish speed and depth distributions did not differ across wall distances, and only the 2.4 cm condition (C7) gave a larger inter-fish distance than 10.4 cm (C6), with Hellinger distance > 0.2. · 4.2 Behavioral response of real fish to the…; 4.2.3 Impact of the virtual fish’s distance to…; 8.2 Captions of the Supplementary Tables; Figure legends
SuggestedReal fish also followed virtual fish on three-petal rose (rhodonea) paths that cross the optically distorted bowl centre. They followed less well when the petals were sharp, and tended to sit on the inner side of the partner's turn.
Peak velocity cross-correlation C ≈ 0.65 for broad-petal Rose 1 (n = 3, d = 5) vs C ≈ 0.51 for sharp-petal Rose 2 (n = 3, d = 1), at lags −0.23 s and −0.1 s; inter-fish distance PDF still peaks at 2.5 cm; speed PDFs centred on 10 cm/s but wider than for circles; perception-angle PDFs asymmetric toward the turn side. 5 one-hour repetitions per rose. · 4.3 Behavioral response of real fish to the…; 8.1 Captions of the Supplementary Figures; Figure legends
Methods, data and reproducibility
One rummy-nose tetra swims freely in a 15 L spherical acrylic bowl, tracked in 3D by an Intel RealSense D435; a C++ Qt simulator drives a virtual fish on fixed circle or rose paths and a Unity renderer projects it from below, anamorphically corrected for the real fish's position. Responses are analysed with Hellinger distances between probability density functions (PDFs) and velocity cross-correlation.
- Control trials: virtual conspecific for 30 min, then a dark virtual sphere for 30 min on the same circular path · Hemigrammus rhodostomus (rummy-nose tetra) · n = 7 one-hour repetitions
- Circular-trajectory trials, 7 conditions varying virtual fish speed, depth or distance to the wall · Hemigrammus rhodostomus (rummy-nose tetra) · n = 10 one-hour repetitions per condition
- Rhodonea (rose curve) trajectory trials, 2 conditions · Hemigrammus rhodostomus (rummy-nose tetra) · n = 5 one-hour repetitions per condition
- Tracking data from all experiments, deposited on figshare · Hemigrammus rhodostomus (rummy-nose tetra) · 10.6084/m9.figshare.29097605
- Methods & tools
- Intel RealSense D435, RealSense SDK, OpenCV, MOG2 background subtraction, Random Matrix Tracker, Kalman filter, Python, Qt (C++), Unity (C#), OPTOMA WUXGA DLP LED projector, Hellinger distance
- Code
- https://doi.org/10.6084/m9.figshare.30188836.v1https://doi.org/10.6084/m9.figshare.30188842.v1https://doi.org/10.6084/m9.figshare.30188845.v1https://doi.org/10.6084/m9.figshare.30188839.v1
- Assay conditions & compute
- One Dell Precision 3640 Tower workstation with an NVIDIA RTX 3070 graphics card handles tracking, trajectory simulation, environment computation and display in real time; tracking runs at 30 to 90 frames per second.
- Reproducibility gaps
- Experimental data are released on figshare, but the text prints only a bare DOI (10.6084/m9.figshare.29097605, listed under methods.data), not a URL. No software or library versions, dependency files or containers are stated, and the number of distinct fish tested is not reported.
Limitations (6)
- Only one real fish and one virtual fish were tested; extending the system to several virtual fish or other species is described as straightforward but is not demonstrated here. · stated by the authors
- Virtual fish followed preset (open-loop) circle and rose paths, and only the anamorphic rendering tracked the real fish; control of the virtual fish by a behavioural model is supported but left for future work. · stated by the authors
- The D435's onboard depth map appears to round some disparities to integers, creating artificial depth bands; the authors' own stereo computation from the IR frames removes this but is noisier and needs Kalman smoothing. · stated by the authors
- Bowl geometry constrains behaviour: a deep virtual fish leaves little vertical room, and paths through the bowl centre are seen through more water, adding optical distortion. · stated by the authors
- The sphere control always came after 30 min with the virtual fish in the same session, so order or habituation effects are not separated from stimulus type; this comparison rests on 7 repetitions. · noted in this summary
- Conditions are compared with pooled distributions and a fixed Hellinger threshold (H > 0.2) rather than tests across individuals; the number of distinct fish is not given, and each rose condition has 5 repetitions. · noted in this summary
Read this if you want to run controlled social-stimulus experiments on freely swimming adult fish and need an open, documented hardware and software stack for 3D tracking and anamorphic projection.
- Citations
- 6
- FWCI
- 23.57
- Field percentile
- top 1%
Abstract
1This study introduces a low-cost, open-source, immersive Virtual Reality (VR) system designed to investigate real-time social interactions and collective behaviors in fish. Understanding collective animal behaviors, such as schooling in fish, presents significant observational challenges due to rapid and complex interactions. To address these difficulties, we developed an innovative closed-loop VR environment allowing precise control and measurement of interactions between real and virtual fish. This setup incorporates high-speed 3D tracking, real-time visual feedback, and automated data processing, creating realistic and interactive scenarios. We present experimental results, obtained with rummy-nose tetras (Hemigrammus rhodostomus ), showing that freely moving real fish consistently adjusted their speed, depth, and spatial positioning to follow virtual fish effectively. Fish matched moderate virtual speeds comfortably, struggled slightly at higher speeds, and actively maintained vertical positioning to sustain group cohesion. The implementation of this VR system provides critical insights into sensory and cognitive processes underlying collective movement and decision-making in fish. By revealing both flexibility and constraints in fish behavior, our findings underscore the potential of VR technology to study social interactions in controlled yet realistic settings. Ultimately, this VR system advances our understanding of collective animal behavior, providing a foundation for cross-species studies linking neural processes, behavioral interactions, and group dynamics.
- organism: zebrafish
- assay: imaging
- research focus: software / tool
- research focus: synaptic & neural circuits
- approach: experimental assay
Reproducibility ledger
Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.
Sign in to file a reproducibility report.
Related papers
- Swimming speed of schooling fish controls social interaction strength in open-loop immersive virtual reality
Ramón Escobedo, Justine Reynaud, … Guy Theraulaz A·Proceedings of the Royal Society B Biological SciencesPeer-reviewed·Jun 3, 2026
- Closed-loop real-virtual interactions validate 3D model of social coordination in fish
Ramón Escobedo, Justine Reynaud, … Guy Theraulaz A·PLoS Computational BiologyPeer-reviewed·Aug 3, 2026
- Validation of a novel immersive Virtual Reality setup with responses of wild-caught freely-moving coral reef fish
M. Vidal, S. C. Mills, … D. Lecchini·bioRxiv·Nov 1, 2022
- Decoding social integration in schooling fish using closed-loop real-virtual interactions
Zhen Kang, Ramón Escobedo, … Guy Theraulaz A·bioRxiv·Mar 18, 2026
- Open-source virtual reality platform for head-fixed adult zebrafish
Alexandre Javier, Sriram Narayanan, … Rainer W. Friedrich·bioRxiv·Oct 24, 2026
- Closed-loop robotic interactions reveal dynamic social filtering in schooling fish
Vaios Papaspyros, Yacine Barhoumi-Andréani, … Guy Theraulaz A·bioRxiv·Aug 5, 2026