Markers of serotonergic function in the orbitofrontal cortex and dorsal raphé nucleus predict individual variation in spatial-discrimination serial reversal learning

Abstract

Dysfunction of the orbitofrontal cortex (OFC) impairs the ability of individuals to flexibly adapt behavior to changing stimulus-reward (S-R) contingencies. Impaired flexibility also results from interventions that alter serotonin (5-HT) and dopamine (DA) transmission in the OFC and dorsomedial striatum (DMS). However, it is unclear whether similar mechanisms underpin naturally occurring variations in behavioral flexibility. In the present study, we used a spatial-discrimination serial reversal procedure to investigate interindividual variability in behavioral flexibility in rats. We show that flexibility on this task is improved following systemic administration of the 5-HT reuptake inhibitor citalopram and by low doses of the DA reuptake inhibitor GBR12909. Rats in the upper quintile of the distribution of perseverative responses during repeated S-R reversals showed significantly reduced levels of the 5-HT metabolite, 5-hydroxy-indoleacetic acid, in the OFC. Additionally, 5-HT2A receptor binding in the OFC of mid- and high-quintile rats was significantly reduced compared with rats in the low-quintile group. These perturbations were accompanied by an increase in the expression of monoamine oxidase-A (MAO-A) and MAO-B in the lateral OFC and by a decrease in the expression of MAO-A, MAO-B, and tryptophan hydroxylase in the dorsal raphé nucleus of highly perseverative rats. We found no evidence of significant differences in markers of DA and 5-HT function in the DMS or MAO expression in the ventral tegmental area of low- vs high-perseverative rats. These findings indicate that diminished serotonergic tone in the OFC may be an endophenotype that predisposes to behavioral inflexibility and other forms of compulsive behavior.

Figure 1

Schematic illustration of the spatial discrimination reversal learning task. Rats were trained under a fixed-ratio (FR) schedule of reinforcement such that three consecutive nose pokes in the same aperture resulted in the delivery of a food pellet in the magazine. A failure to respond within 30 s (an ‘omission') resulted in a 5 s time-out period. Following the acquisition of a spatial discrimination, the contingency was reversed such that responses in the previously incorrect aperture were now correct (and vice versa). Animals completed three reversals (‘ × 3') in a single 1 h session.

Figure 2

Coronal (a) and sagittal (b) sections showing the regions of interest used for neurochemical assessment in post-mortem tissue of rats stratified according to low-, mid-, and high-perseverative behavior on a spatial discrimination serial reversal task. Adapted from Paxinos and Watson (1998).

Figure 3

Cumulative frequencies of perseverative responses, total trials, and errors to reach criterion, during initial testing, in all four cohorts. Cohort 1: systemic drug administration (n=48); cohort 2: HPLC-ECD analysis (n=44); cohort 3: autoradiography (n=44); and cohort 4: qRT-PCR (n=47).

Figure 4

Effect of citalopram (n=24) on (a) the number of trials to criterion; (c) incorrect trials to criterion; and (e) percentage perseverative responses. Effect of GBR 12909 (n=23) on (b) the number of trials to criterion; (d) incorrect trials to criterion; and (f) percentage perseverative responses on the spatial reversal learning task. Data are means±SEM from a single reversal learning session. Asterisks denote a significant difference between the groups indicated: *p<0.05 and **p<0.01.

Figure 5

(a) Levels of 5-HT and (b) 5-HIAA (pmol/mg) in the medial and lateral orbitofrontal cortex (OFC) of high- (n=9), mid- (n=10), and low- (n=9) perseverative animals. (c) 5-HT_2A and (d) 5-HTT receptor binding in the medial and lateral OFC of high- (n=9), mid- (n=10), and low- (n=15) perseverative groups. Data are means±1 SEM. Asterisks denote a significant difference between the groups indicated: *p<0.05 and ⁺p=0.059.

Figure 6

Expression of (a) tph2 and slc4a6 in the dorsal raphé nucleus (DRN) and (b) th and slc3a6 in the ventral tegmental area (VTA) of high- (n=8), mid- (n=8), and low- (n=9) perseverative groups. Data are means±1 SEM. Asterisks denote a significant difference between the groups indicated: *p<0.05.

Figure 7

Expression of (a) MAO-A and (b) -B in the dorsal raphé nucleus (DRN) and ventral tegmental area (VTA) of high- (n=8), mid- (n=8), and low- (n=9) perseverative groups. (c) MAO-A and (d) -B in the medial and lateral OFC of high- (n=8), mid- (n=10) and low- (n=9) perseverative groups. Data are means±1 SEM. Asterisks denote a significant difference between the groups indicated: *p<0.05 and **p<0.01.