Learning to Express Reward Prediction Error-like Dopaminergic Activity Requires Plastic Representations of Time

Abstract

The dominant theoretical framework to account for reinforcement learning in the brain is temporal difference (TD) reinforcement learning. The normative motivation for TD theory is that the brain needs to learn about expected future rewards in order to learn how to maximize these rewards. The TD framework predicts that some neuronal elements should represent the reward prediction error (RPE), which means they signal the difference between the expected future rewards and the actual rewards. What makes the TD learning theory so prominent is that the firing properties of dopaminergic neurons in the ventral tegmental area (VTA) appear similar to those of RPE model-neurons in TD learning. Biologically plausible implementations of TD learning assume a fixed temporal basis for each stimulus that might eventually predict a reward. Here we show on the basis of first principles that such a fixed temporal basis is implausible. We also show that certain predictions of TD learning are inconsistent with experimental data. We propose instead an alternative theoretical framework, coined FLEX (Flexibly Learned Errors in Expected Reward). In FLEX, feature specific representations of time are learned, allowing for neural representations of stimuli to adjust their timing and relation to rewards in an online manner. As an indirect consequence, dopamine in FLEX resembles, but is not equivalent to RPE. In FLEX dopamine acts as an instructive signal which helps build temporal models of the environment. FLEX is a general theoretical framework that has many possible biophysical implementations. In order to show that FLEX is a feasible approach, we present a specific biophysically plausible model which implements the principles of FLEX. We show that this implementation can account for various reinforcement learning paradigms, and that its results and predictions are consistent with a preponderance of both existing and reanalyzed experimental data.