Combining Unity with machine vision to create low latency, flexible, and simple virtual realities

Abstract

O_LIIn recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviors. By using the actions of a constrained animal to control the visual scenery, the animal is provided the perception of moving through a simulated environment. As the animal is constrained in space, this allows detailed behavioral quantification. Additionally, as the world is generally computer-generated this allows for mechanistic quantification of visual triggers of behavior. C_LIO_LIWe created a novel virtual arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi-modal virtual reality arena, MultiMoVR (Kaushik et al., 2020) but tracked hoverfly wing movements using DeepLabCut-live (DLC-live, Kane et al., 2020). For trackball experiments, we recorded the motion of a ball that a tethered crab was walking on using FicTrac (Moore et al., 2014). In both cases, real-time tracking was interfaced with Unity to control the movement of the tethered animals avatars in the virtual world. We developed a user-friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. C_LIO_LIWe show that both the DLC-live-Unity and the FicTrac-Unity configurations close the feedback loop effectively with small delays, less than 50 ms. Our FicTrac-Unity integration highlighted the importance of closed-loop feedback by reducing behavioral artifacts exhibited by the crabs in open-loop scenarios. We show that Eristalis tenax hoverflies, using the DLC-live-Unity integration, navigate towards flowers. The effectiveness of our CAVE interface is shown by implementing experimental sequencing control based on avatar proximity to virtual structures. C_LIO_LIOur results show that combining Unity with machine vision tools such as DLC-live and FicTrac provides an easy and flexible virtual reality (VR) environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design and implement new experiments without programming in code. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community. C_LI