The Neural Architecture of Theory-based Reinforcement Learning

Abstract

Humans learn internal models of the environment that support efficient planning and flexible generalization in complex, real-world domains. Yet it remains unclear how such internal models are represented and learned in the brain. We approach this question within the framework of theory-based reinforcement learning, a strong form of model-based reinforcement learning in which the model is an intuitive theory - a rich, abstract, causal model of the environment built on a natural ontology of physical objects, intentional agents, relations, and goals. We used a theory-based reinforcement learning model to analyze brain data from human participants learning to play different Atari-style video games while undergoing functional MRI. Theories inferred by the theory-based model explained the signal in inferior frontal gyrus and other prefrontal areas better than several alternative models. Brain activity increased in response to theory update events in inferior frontal gyrus, occipital cortex, and fusiform gyrus, with separate learning signals for different theory components. This corresponded with a transient strengthening of theory representations in those regions. Finally, the effective connectivity pattern during theory updating suggests that information flows top-down from theory-coding regions in the prefrontal cortex to theory updating regions in occipital and temporal cortex. These results are consistent with a neural architecture in which top-down theory representations originating in prefrontal regions shape sensory predictions in visual areas, where factorized theory prediction errors are computed and in turn trigger bottom-up updates of the theory. This initial sketch provides a foundation for understanding of the neural representations and computations that support efficient theory-based reinforcement learning in complex, naturalistic environments.