5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence

Abstract

Rationale: Cognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.

Objectives: We investigated the effects of 5-HT_2A receptor (5-HT_2AR) and 5-HT_2C receptor (5-HT_2CR) antagonism on flexibility and underlying RL mechanisms.

Methods: Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT_2AR and 5-HT_2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.

Results: 5-HT_2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT_2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT_2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT_2CR antagonism also decreased beta, but reduced side stickiness.

Conclusions: These data indicate that 5-HT_2A and 5-HT_2CRs both modulate different aspects of flexibility, with 5-HT_2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT_2CRs for learning from positive feedback assessed through conventional measures.

Keywords: 5-HT receptors; Cognitive flexibility; Punishment learning; Reinforcement learning; Reward learning; Stickiness.

Fig. 1

Experimental design. (A) Table of groups and treatments. (N, number of subjects). (B) VPVD stages and stimuli in the M100907 and SB-242,084 experiments. A is the 100% reinforced stimulus, B is the 0% reinforced stimulus, C is reinforced on 50% of probe trials. (C) Experimental timeline, including pretraining and experimental weeks. (disc., discrimination; VPVD, valence-probe visual discrimination (1 and 2); VS, visual stimulus pair (1, 2 and 3); VS2, visual stimulus pair 2; VS3, visual stimulus pair 3

Fig. 2

Effects of M100907 on VPVD parameters. (A) Percent correct and percent optimal choice across sessions. (B) Errors to criterion and errors per phase. Results are represented as mean ± standard error of the mean (SEM); *** p < 0.01, # p < 0.1

Fig. 3

Results from the hierarchical Bayesian winning RL model 9, showing differences in group mean parameters following M100907 administration. (A) Reward and punishment learning rate parameters. (B) Reinforcement sensitivity, side and stimulus stickiness parameters. (C) Discount factor. (LOW, low dose; HIGH, high dose; CON, vehicle; Reinf., reinforcement; HDI, highest posterior density interval. Red indicates 0 ∉ 95% HDI; orange indicates 0 ∉75% HDI)

Fig. 4

Effects of SB-242048 on VPVD parameters. (A) Percent correct and percent optimal choice across sessions. (B) Errors to criterion and errors per phase. Results are represented as mean ± SEM; *** p < 0.01, # p < 0.1

Fig. 5

Results from the hierarchical Bayesian winning RL model 7, showing differences in group mean parameters following SB-242048 administration. (A) Reward and punishment learning rate parameters. (B) Reinforcement sensitivity and side stickiness parameters. (LOW, low dose; HIGH, high dose; CON, vehicle; Reinf., reinforcement; HDI, highest posterior interval. Red indicates 0 ∉ 95% HDI; orange indicates 0 ∉ 75% HDI)