Dopamine-independent state inference mediates expert reward guided decision making

Abstract

Rewards are thought to influence future choices through dopaminergic reward prediction errors (RPEs) updating stored value estimates. However, accumulating evidence suggests that inference about hidden states of the environment may underlie much adaptive behaviour, and it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about recent reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model in which frontal cortex learned to track hidden task states by predicting observations, while basal ganglia learned corresponding values and actions via dopaminergic RPEs. Together, this two-process account reconciles how dopamine-independent state inference and dopamine-mediated reinforcement learning interact on different timescales to determine reward-guided choices.