Understanding neural circuit principles for representation learning through joint-embedding predictive architectures

Abstract

Whether tracking prey or evading a predator, an animal's survival depends on both identifying objects in the environment and anticipating how they will move. To guide behavior, the brain must therefore extract information about object identities and their dynamics from entangled sensory inputs. How it accomplishes this feat remains an open question. Classical predictive coding theories propose that this ability arises by comparing predicted sensory signals with actual inputs and reducing the associated prediction errors. While such models capture important aspects of cortical computation, they typically focus on faithfully predicting sensory input and do not explicitly address how abstract, untangled representations of objects and their dynamics emerge solely through experience. Here, we develop a theory of representation learning in neural circuits that shifts the focus from prediction in the input space to prediction in representation space, without relying on external supervision or labeled data. Specifically, we introduce recurrent predictive learning (RPL), a recurrent joint-embedding predictive architecture inspired by self-supervised learning, that learns abstract representations of object identity and their dynamics and predicts future object motion from continuous sensory streams. While local learning rules alone can learn invariant object identity, we find that capturing object dynamics requires a circuit that explicitly generates and minimizes internal prediction-error signals. Crucially, the model learns sequence representations that resemble successor-like representations observed in the human primary visual cortex. The model also develops abstract sequence representations comparable to those reported in the macaque prefrontal cortex. Finally, we demonstrate a hierarchical version of RPL and discuss possible mappings onto cortical microcircuits. Our work establishes a circuit-centric theory framework that provides new perspectives on how the brain may acquire an internal model of the world through experience.