Vector-valued dopamine improves learning of continuous outputs in the striatum

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatumwide RPE signal can support RL over a small discrete set of actions (e.g. no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions, but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of dynamic outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). Using a recurrent network model of the basal ganglia, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. The corticostriatal plasticity rule we employed is based on Random Feedback Learning Online learning and is a fully local, "three-factor" product of the presynaptic firing rate, a post-synaptic factor and the unique dopamine concentration perceived by each striatal neuron. Crucially, we demonstrate that under this plasticity rule, the improvement in learning does not require precise nigrostriatal synapses, but is compatible with random placement of varicosities and diffuse volume transmission of dopamine.