henhaobio
Sign in

Scientific ReportsAnimal Behavior & CognitionPeer-reviewed journal article

Uncertainty and reward histories have distinct effects on decisions after wins and losses

Shivam KalhaniD, Robin MagnardiD, Zhenlong Zhang, Yifeng ChengiD, Patricia H. JanakiD

PDFDOIPreprint
In-depth full-text digest available · Jump to digest ↓

Abstract

Intelligent behavior necessitates an adaptive integration of feedback. It is well-known that animals asymmetrically learn from positive and negative feedback. While asymmetrical learning is a robust behavioral effect, the latent computations behind how animals represent their environments and use this to differentially weight wins and losses is poorly understood. Here we tested whether and how uncertainty and reward histories modulate the weights placed on wins and losses using a behavioral data set collected in rats. We propose a reinforcement learning model that integrates uncertainty history via an unsigned average reward prediction error and a separate subjective reward history component. We showed that in a dynamic probabilistic reversal learning task with blocks of variable reward predictability, ongoing estimation of uncertainty history and reward history both distinctly influenced rats sensitivity to wins and losses. In more predictable environments, and under low uncertainty levels, i.e., when rats were certain in making correct choices, rats weighted wins more than losses, as indicated by a higher win-stay, and lower lose-shift probability. This asymmetrical learning strategy enabled rats to remain with the correct action, while discounting the influence of rare losses. Further, male rats were more impacted by their reward history, i.e., environmental richness, when making lose-shift decisions, but conversely, female rats were more influenced by their uncertainty history. Hence, we found sex-specific contributions of these latent computations in modulating behavior. We overall demonstrate that asymmetrically weighting wins and losses could form an important behavioral strategy when adapting to ongoing changes in reward and uncertainty history.

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (19 Aug 2025) · CC BY 4.0

In 14 rats doing probabilistic reversal learning, model-derived uncertainty (unsigned average prediction error) and recency-weighted reward history distinctly and sex-specifically shaped win-stay versus lose-shift choices.

Animals learn more from wins than losses, yet which environmental statistics drive this asymmetry is unclear, since prior adaptive-learning-rate models did not let uncertainty and reward history separately bias positive versus negative feedback. This work tests an unsigned average reward prediction error (avgRPE, uncertainty history) and a recency-weighted global reward state (GRS, reward history) as modulators of win and loss sensitivity, including sex differences.

Key results

  1. ShownRats weighted wins over losses overall, with higher win-stay (WS) than lose-shift (LS) probability; males were more win-sensitive and less loss-sensitive than females, which hierarchical modeling linked mainly to stronger unchosen-action value decay after losses in males.

    WS vs LS: F(1,24) = 907.28, p = 2.2e-16; WS/LS by sex F(1,24) = 14.87, p = 7.6e-04; males higher WS (p = 0.006, Cohen's d = 1.70), lower LS (p = 0.02, d = 1.35). Directed Bayes factors (males vs females): α+ 11.31, α- 4.47, decay after losses 20.98, decay after wins 1.04. · 1. Rats were more sensitive to wins and less…; Figure legends

  2. ShownWin-stay rose from early to late block phase while lose-shift did not; late in blocks, the higher-probability ('better') lever drew more win-stay and less lose-shift than the worse lever, especially in high-contrast blocks, where males suppressed better-lever lose-shift more than females.

    Late vs early: WS p = 0.0001 (d = 2.04), LS p = 0.44. Better vs worse: WS F(1,36) = 57.53, p = 5.68e-09; LS F(1,36) = 436.29, p = 2.20e-12. High-contrast action differences larger: WS F(1,13) = 22.65, p = 0.0004; LS F(1,13) = 175.92, p = 6.24e-09. Sex gap in better-lever LS, high contrast: p = 0.0006, d = 3.30. · 2. Asymmetrical learning strategies enabled rats…; Figure legends

  3. ShownChoices drew on prior-trial as well as current prediction errors; a model using an exponentially weighted average of RPEs beat a standard RL model, the unsigned value of that average rose with block stochasticity (an uncertainty proxy), and GRS mirrored empirical reward rate across blocks.

    Current-vs-previous RPE weight k near 0.82 (WS) and 0.72 (LS) correlated most strongly with choices; males leaned more on the prior RPE for LS (p < 1e-04, d = 3.37). avgRPE model had better BIC than standard RL. Unsigned avgRPE differed by block (F = 379.6, p = 2.2e-16): lowest high contrast, highest no contrast. · 3. Using unsigned avgRPE as a proxy for the…; 4.3. Model-dependent correlation analyses with…; Figure legends

  4. SuggestedUncertainty and reward history acted differently on better-lever late-phase choices: low avgRPE favored win-stay mainly in high-contrast blocks and more so in males, high GRS favored win-stay regardless of block or sex, and for lose-shift females tracked avgRPE while males tracked GRS more.

    avgRPE-WS: block F(1,12) = 106.29, p = 2.57e-07; sex by block F(1,12) = 8.52, p = 0.013 (males, high contrast p = 1e-04, d = 2.77). GRS-WS: block F(1,36) = 0.03, p = 0.87; sex F(1,36) = 0.90, p = 0.36. LS: stronger avgRPE link in females (p = 0.02, d = 1.5); stronger GRS link in males, high contrast (p = 0.025, d = 2.52). · 4. Uncertainty and reward history differentially…; Figure legends

  5. SuggestedIn a median-split analysis, avgRPE and GRS jointly governed win-stay only in males in high-contrast blocks, where lowering win-stay required both low GRS and high avgRPE; in females, GRS alone shifted win-stay irrespective of uncertainty or block.

    Males: sub-group F(3,56) = 19.21, p = 1.09e-08; block F(1,36) = 8.39, p = 0.005; interaction F(3,56) = 3.77, p = 0.016; low GRS plus high avgRPE needed to lower WS (p = 0.0001, d = 2.40). Females: sub-group F(3,28) = 8.50, p = 0.0004; block F(1,28) = 0.61, p = 0.44; interaction F(3,28) = 0.23, p = 0.89. · 5. The uncertainty state and the GRS interact to…; Figure legends

Methods, data and reproducibility

Reanalysis of win-stay/lose-shift choices by rats in a dynamic probabilistic reversal task with high-, low- and no-contrast blocks, fitting standard, average-RPE and global reward state reinforcement learning models plus a five-parameter asymmetric model (hierarchical Bayesian, by sex), and relating latent estimates to choices via mixed models.

  • Water-restricted Long-Evans rats (non-ethanol-exposed subset of the authors' earlier study) on the dynamic probabilistic reversal learning (dynaPRL) task, average of 30 sessions · rat (Long-Evans) · n = 14 (9 males, 5 females)
  • Full Long-Evans cohort trained on the standard probabilistic reversal learning task (70% vs 10% reward; 15 two-hour sessions), reported in Figure S1 · rat (Long-Evans) · n = 28 total (18 males, 10 females)
Methods & tools
MATLAB 2024b (fmincon, session-level maximum likelihood), Stan via the MatlabStan interface (Hamiltonian Monte Carlo), Bayesian information criterion (BIC) model comparison, Directed Bayes factor on group-difference hyperparameters, RStudio 4.1.2 (lmer linear mixed-effects models), emmeans post-hoc tests with Tukey correction, Binomial logistic regression, Standard RL model with softmax choice, avgRPE RL model (history weight hW), Global reward state (GRS) model, Five-parameter asymmetric RL model (α+, α-, γ+, γ-, β), Med Associates operant chambers
Reagents & environment
MATLAB 2024b (fmincon); Stan through the MatlabStan interface; RStudio 4.1.2 (lmer, emmeans)
Reproducibility gaps
Data are promised for the Open Science Framework only upon publication, with no link or identifier given; no availability of analysis or modeling code is stated.
Limitations (4)
  • Late-phase lose-shift trials after better-lever choices were too scarce, especially in high-contrast blocks, so the GRS-by-uncertainty interaction could be tested for win-stay only. · stated by the authors
  • Males here were more exploitative, opposite to earlier mouse bandit work where females exploited more; the authors suggest species, the reversal design, or studying post-acquisition performance rather than acquisition as possible reasons, leaving it unresolved. · stated by the authors
  • Sex effects rest on 9 males and 5 females from a single reanalyzed cohort without replication, and avgRPE and GRS are model-derived proxies related to choices correlationally, so mechanistic claims remain untested. · noted in this summary
  • Internal inconsistencies: low-contrast block given as 60% vs 30% (Results, Figure 1) but 60% vs 10% (Methods); a positive avgRPE-LS correlation is called 'stronger negative' for females; the male GRS-LS effect is assigned to different blocks in Results and Figure 6. · noted in this summary

Read this if you model reward-guided choice or reversal learning in rodents and want a way to separate uncertainty from reward-history effects on win-stay/lose-shift behavior, including sex differences.

Citations
4
FWCI
16.72
Field percentile
top 1%

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISShahdoust '26Harris '20Sun '22Woo '23Ganesh '24Bruckner '20
76% semantic matchNeuroscience
Niloufar Shahdoust, Rhiannon L Cowan, T. Alexander Price+9 more·bioRxiv·Mar 11, 2026