A nonlinear relationship between prediction errors and learning rates in human reinforcement learning

Abstract

Reinforcement-learning (RL) models have been pivotal to our understanding of how agents perform learning-based adaptions in dynamically changing environments. However, the exact nature of the relationship (e.g. linear, logarithmic etc.) between key components of RL models such as prediction errors (PEs; the difference between the agents expectation and the actual outcome) and learning rates (LRs; a coefficient used by agents to update their beliefs about the environment) has not been studied in detail. Here, across (i) simulations, (ii) reanalyses of readily available datasets and (iii) a novel experiment, we demonstrate that the relationship between PEs and LRs is (i) nonlinear over the PE/LR space, and (ii) it can be better accounted for by an exponential-logarithmic function that can transform the magnitude of PEs instantaneously to LRs. In line with the temporal predictions of this model, we show that physiological correlates of LRs accumulate while learners observe the outcome of their choices and update their beliefs about the environment.