Opponent learning with different representations in the cortico-basal ganglia circuits

Abstract

The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such combined use of different representations coupled with appetitive or aversive learning has computational benefits. We simulated reward learning tasks in dynamic environments, and examined the performance of animal modeled as an agent equipped with two learning systems, each of which used individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative reward prediction errors (RPEs) in each system, we found that combination of an SR-based system learning mainly from positive RPEs and an IR-based system learning mainly from negative RPEs could achieve good performance, as compared to other combinations, in many situations. The architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that combining different representations with appetitive and aversive learning is an effective learning strategy adopted by the brain.