Generalizing to generalize: humans flexibly switch between compositional and conjunctive structures during reinforcement learning

Abstract

Humans routinely face novel environments in which they have to generalize in order toact adaptively. However, doing so involves the non-trivial challenge of deciding which aspects of a task domain to generalize. While it is sometimes appropriate to simply re-use a learned behavior, often adaptive generalization entails recombining distinct components of knowledge acquired across multiple contexts. Theoretical work has suggested a computational trade-off in which it can be more or less useful to learn and generalize aspects of task structure jointly or compositionally, depending on previous task statistics, but empirical studies are lacking. Here we develop a series of navigation tasks which manipulate the statistics of goal values ("what to do") and state transitions ("how to do it") across contexts, and assess whether human subjects generalize these task components separately or conjunctively. We find that human generalization is sensitive to the statistics of the previously experienced task domain, favoring compositional or conjunctive generalization when the task statistics are indicative of such structures, and a mixture of the two when they are more ambiguous. These results support the predictions of a normative "meta-generalization learning" agent that does not only generalize previous knowledge but also generalizes the statistical structure most likely to support generalization. Author NoteThis work was supported in part by the National Science Foundation Proposal 1460604 "How Prefrontal Cortex Augments Reinforcement Learning" to MJF. We thank Mark Ho for providing code used in the behavioral task. We thank Matt Nassar for helpful discussions. Correspondence should be addressed to Nicholas T. Franklin (nfranklin@fas.harvard.edu) or Michael J. Frank (michael_frank@brown.edu).