Driving singing behaviour in songbirds using multi-modal, multi-agent virtual reality

Abstract

Interactive biorobotics provides unique experimental potential to study the mechanisms underlying social communication but is limited by our ability to build expressive robots that exhibit the complex behaviours of birds and small mammals. An alternative to physical robots is to use virtual reality (VR). Here, we designed and built a modular, audio-visual virtual reality environment that allows online, multi-modal, multi-agent interaction for social communication. We tested this system in songbirds, which provide an exceptionally powerful and tractable model system to study social communication. We show that zebra finches (Taeniopygia guttata) communicating through the VR environment exhibit normal call timing behaviour, males sing female directed song and both males and females display high-intensity courtship behaviours to their mates. These results suggest that the VR system provides a sufficiently natural environment to elicit normal social communication behaviour. Furthermore, we developed a fully unsupervised online song motif detector and used it to manipulate the virtual social environment of male zebra finches based on the number of motifs sung. Our VR setup represents a first step in taking automatic behaviour annotation into the online domain and allows for animal-computer interaction using higher level behaviours such as song. Our unsupervised acoustic analysis eliminates the need for annotated training data thus reducing labour investment and experimenter bias.