Extinction burst could be explained by curiosity-driven reinforcement learning

Abstract

Curiosity encourages agents to explore their environment, leading to learning opportunities. Although psychology and neurobiology have tackled how external rewards control behavior, how intrinsic factors control behavior remains unclear. An extinction burst is a behavioral phenomenon in which a sudden increase in the frequency of a behavior immediately follows the omission of a reward. Although the extinction burst is textbook knowledge in psychology, there is little empirical evidence of it in experimental situations. In this study, we show that the extinction burst can be explained by curiosity by combining computational modeling of behavior and empirical demonstrations in mice. First, we built a reinforcement learning model incorporating curiosity, defined as expected reward prediction errors, and the model additively controlled the agents behavior to the primary reward. Simulations revealed that the curiosity-driven reinforcement learning model produced an extinction burst and burst intensity depended on the reward probability. Second, we established a behavioral procedure that captured extinction bursts in an experimental setup using mice. We conducted an operant conditioning task with head-fixed mice, in which the reward followed after pressing a lever at a given probability. After the training sessions, we occasionally withheld the reward delivery when the mice performed the task. We found that phasic bursts of responses occurred immediately after reward omission when responses were rewarded with a high probability, suggesting that the magnitude of reward prediction errors controlled the burst. These results provide theoretical and experimental evidence that intrinsic factors control behavior in adapting to an ever-changing environment. Significance statementIn theories of learning and behavior, primary rewards such as food, water, and sex occupy a dominant position as factors controlling behavior. However, primary rewards are scarce. Experimental investigations in psychology, neuroscience, ethology, and economics have revealed that novelty, uncertainty, and unpredictability drive behavior. How these intrinsic factors affect behavior is essential for a comprehensive understanding of the principles of learning and behavior. This study provides theoretical and experimental evidence that operant responses in mice are directly controlled by external rewards and intrinsic factors such as curiosity. Our study provides a robust example of curiosity-driven behavior and paves the way for understanding the mechanism of curiosity.