5-HT 2A and 5-HT 2C receptor antagonism differentially modulates reinforcement learning and cognitive flexibility: behavioral and computational evidence

Cognitive flexibility, the ability to adapt behavior in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder (OCD) and major depressive disorder (MDD). Evidence suggests that flexibility, which can be operationalized using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behavior and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown. We investigated the effects of 5-HT2A receptor (5-HT2AR) and 5-HT2C receptor (5-HT2CR) antagonism on cognitive flexibility and underlying RL mechanisms. Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2AR and 5-HT2CR antagonists M100907 and SB-242084, respectively, on reversal learning performance and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task. 5-HT2AR antagonism impaired reversal learning during certain phases. 5-HT2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2AR antagonism decreased punishment learning rate at high and low doses. The low dose also increased exploration (beta) and increased stimulus and side stickiness (kappa). 5-HT2CR antagonism also increased beta, but reduced side stickiness. These data indicate that 5-HT2A and 5-HT2CRs both modulate different aspects of flexibility, with 5-HT2ARs modulating learning from negative feedback and 5-HT2cRs for learning from positive feedback.