Even if suboptimal, novelty drives human exploration

Abstract

Humans successfully explore their environment to find extrinsic rewards, even when exploration requires several intermediate reward-free decisions. It has been hypothesized that intrinsic rewards such as novelty guide this reward-free exploration. However, different intrinsic rewards lead to different exploration strategies, some prone to suboptimal attraction to irrelevant stochastic stimuli, sometimes called the noisy TV problem. Here, we ask whether humans show a similar attraction to reward-free stochasticity and, if so, which type of intrinsic reward guides their exploration. We design a multi-step decision-making paradigm where human participants search for rewarding states in an environment with a highly stochastic but reward-free sub-region. We show that (i) participants persistently explore the stochastic sub-region and (ii) their decisions are best explained by algorithms driven by novelty but not by optimal algorithms driven by information gain. Our results suggest that humans use suboptimal but computationally cheap strategies for exploration in complex environments.