Adaptive Reinforcement Learning is causally supported by Anterior Cingulate Cortex and Striatum

Abstract

Learning the reward structure of complex environments can be achieved using reinforcement learning processes augmented with cognitive strategies. Among these strategies is the adjustment of the exploration-exploitation trade-off to increase exploration when behavior gets stuck and increase exploitation when reward contingencies remain stable. Here we tested how the anterior cingulate cortex (ACC) and the striatum causally support adaptive cognitive strategies augmenting reinforcement learning. We electrically microstimulated the ACC or the striatum in nonhuman primates at the time they chose multidimensional objects to learn about their reward values, while varying target feature uncertainty and the motivational saliency of the chosen objects. We found that stimulation of the ACC and the striatum affected adaptive strategies and reinforcement learning when target feature uncertainty was high, but in opposite ways. ACC-stimulation impaired learning and sustaining correct responses, while striatum-stimulation on average improved learning from rewarding outcomes. Behavioral modeling showed that stimulation affected the same mechanisms but in opposite ways. ACC stimulation impaired the monitoring of outcome uncertainty for adapting exploration-exploitation and reduced the ability to lower prediction errors during learning, while striatum stimulation enhanced the monitoring of outcome uncertainty and prediction-error based updating of object values. Stimulation did not alter the use of working memory or attentional filtering as alternative learning strategy. These opposing behavioral stimulation effects were associated with the ACC having populations of neurons that fired stronger during choices that were more uncertain, had a lower value, and that tracked error history, while striatum neurons more likely encoded higher values and more certain choices. In summary, microstimulation during object choices suggest that ACC and the striatum causally adapt exploration-exploitation levels to guide exploration towards reward-relevant objects during periods of uncertainty. Short SummaryThe anterior cingulate cortex (ACC) and the striatum are core nodes of a network supporting reinforcement learning, but how they augment reinforcement learning with adaptive cognitive strategies has remained unresolved. This study uses electrical microstimulation to show that ACC and the striatum have causal roles adjusting the exploration-exploitation balance and optimize reinforcement learning in multidimensional environments, while not altering attentional filtering or working memory strategies.