On the normative advantages of dopamine and striatal opponency for learning and choice

The basal ganglia (BG) contribute to reward-driven behavior, but it is unclear why such behavior should rely on notoriously complex circuitry involving opponent striatal pathways. Dopamine (DA) nonlinearly modulates activity and plasticity in the BG and is itself dynamically modulated across tasks. We develop the OpAL* model to assess the normative advantages of such circuitry. In OpAL*, dynamic DA modulation differentially amplifies the striatal pathway most tuned to the accumulated history of positive or negative outcomes. This efficient coding mechanism avoids a vexing explore-exploit tradeoff that plagues traditional RL models in sparse reward environments. OpAL* exhibits robust advantages over alternative models across a range of environments, particularly with sparse reward. These advantages depend on opponent and nonlinear Hebbian plasticity mechanisms previously thought to be pathological. Finally, OpAL* captures risky choice patterns arising from DA and environmental manipulations across species, suggesting that they result from a normative biological mechanism. Everybody wants the most they can possibly get For the least they can possibly do - Todd Snider, "Easy Money"