Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Abstract

Animals have a remarkable ability to act autonomously---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. To understand how biological autonomy may emerge from optimizing intrinsic goals, we use recent whole-brain recordings in the fruit fly (\textit{Drosophila melanogaster}) that identify neural representations of spontaneous exploratory decisions more than $10$ seconds in advance. Combined with a neuromechanical digital twin of \textit{Drosophila}, these recordings allow us to test whether intrinsically motivated artificial agents with realistic embodiments quantitatively match fly behavior and internal representations of future volitional action during exploration. We implement a hierarchical, intrinsically motivated agent whose policy operates over a discrete library of motor primitives---fixed low-level controllers that actuate joint dynamics---closely matching ethological theories of behavioral syllables. We train agents with no external reward using $20$ intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. We evaluate frozen policies in a virtual tethered-ball assay modeled after the fruit-fly experiment, allowing us to compare spontaneous locomotor statistics and internal representations and decode future turn direction from both kinematics and neural activations on the same timescales observed in flies. We find that DIAYN---a hierarchical competency algorithm that learns a discrete set of policies maximizing behavioral entropy and the separability of their induced sensory observations---achieves the strongest joint behavioral and neural prospective coding performance with the fruit fly. Surprisingly, DIAYN's learned policies closely capture individual variability in fly exploration strategies, suggesting that this behavioral diversity optimizes a concrete computational goal of maximizing state-space coverage at the \emph{population} level. Furthermore, DIAYN agents' latent activity closely captures the specific sum-difference rule in interhemispheric encoding of future kinematics in biological flies. Together, these results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.