Reinforcement-based option competition in human dorsal stream during exploration/exploitation of a continuous space

Abstract

Primates exploring and exploiting a continuous sensorimotor space rely on maps in the dorsal stream that guide visual search, locomotion, and grasp. For example, an animal swinging from one tree limb to the next uses rapidly evolving sensorimotor representations to decide when to harvest a reward. We show that such exploration/exploitation depends on dynamic maps of competing option values in the human dorsal stream. Using a reinforcement learning (RL) model capable of rapid learning and efficient exploration and exploitation, we show that preferred options are selectively maintained on the map while the values of spatiotemporally distant alternatives are compressed. Consistent with biophysical models of cortical option competition, dorsal stream BOLD signal increased and posterior cortical {beta}1/ oscillations desynchronized as the number of potentially valuable options grew, matching predictions of information-compressing RL rather than traditional RL that caches long-term values. BOLD and {beta}1/ responses were correlated and predicted the successful transition from exploration to exploitation. These option competition dynamics were observed across parietal and frontal dorsal stream regions, but not in the occipito-temporal MT+ sensitive to the average reward rate. Our results also illustrate that models diverging predictions about information dynamics can help to adjudicate between them based on population activity. Graphical abstract O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=148 SRC="FIGDIR/small/541828v1_ufig1.gif" ALT="Figure 1"> View larger version (74K): org.highwire.dtl.DTLVardef@155b3c8org.highwire.dtl.DTLVardef@18e28c5org.highwire.dtl.DTLVardef@16d7c1org.highwire.dtl.DTLVardef@1aed684_HPS_FORMAT_FIGEXP M_FIG C_FIG