A network model of basal ganglia for understanding the roles of dopamine and serotonin in reward-punishment-risk based decision making

Abstract

There is significant evidence that in addition to reward-punishment based decision making, the Basal Ganglia (BG) contributes to risk-based decision making (Balasubramani et al., 2014). Despite this evidence, little is known about the computational principles and neural correlates of risk computation in this subcortical system. We have previously proposed a reinforcement learning (RL)-based model of the BG that simulates the interactions between dopamine (DA) and serotonin (5HT) in a diverse set of experimental studies including reward, punishment and risk based decision making (Balasubramani et al., 2014). Starting with the classical idea that the activity of mesencephalic DA represents reward prediction error, the model posits that serotoninergic activity in the striatum controls risk-prediction error. Our prior model of the BG was an abstract model that did not incorporate anatomical and cellular-level data. In this work, we expand the earlier model into a detailed network model of the BG and demonstrate the joint contributions of DA-5HT in risk and reward-punishment sensitivity. At the core of the proposed network model is the following insight regarding cellular correlates of value and risk computation. Just as DA D1 receptor (D1R) expressing medium spiny neurons (MSNs) of the striatum were thought to be the neural substrates for value computation, we propose that DA D1R and D2R co-expressing MSNs are capable of computing risk. Though the existence of MSNs that co-express D1R and D2R are reported by various experimental studies, prior existing computational models did not include them. Ours is the first model that accounts for the computational possibilities of these co-expressing D1R-D2R MSNs, and describes how DA and 5HT mediate activity in these classes of neurons (D1R-, D2R-, D1R-D2R- MSNs). Starting from the assumption that 5HT modulates all MSNs, our study predicts significant modulatory effects of 5HT on D2R and co-expressing D1R-D2R MSNs which in turn explains the multifarious functions of 5HT in the BG. The experiments simulated in the present study relates 5HT to risk sensitivity and reward-punishment learning. Furthermore, our model is shown to capture reward-punishment and risk based decision making impairment in Parkinson's Disease (PD). The model predicts that optimizing 5HT levels along with DA medications might be essential for improving the patients' reward-punishment learning deficits.

Keywords: D1 and D2 receptor co-expression; basal ganglia network; dopamine; medium spiny neurons; punishment; reward; risk; serotonin.

Figure 1

(A) Schematic of the cellular correlate model for the value and the risk computation in the striatum, (B) The D1, D2, and D1D2 gain functions, (C) The output activity of D1R MSN (yD1), D1R-D2R co-expressing MSN (yD1D2), variance tracked through Equation (2.1.5) containing δ2, and normalized variance computed analytically (var) = p^*(1-p); Here p is the probability associated with rewards, i.e., with probability p, reward = 1, else reward = 0. The resemblance of var to yD1D2 shows the ability of D1R-D2R co-expressing MSN to perform risk computation.

Figure 2

The schematic flow of the signal in the network model. Here x denotes the presence of a state; a denotes the action; with the subscript denoting the index i; Since most of the experiments in the study simulate two possible actions for any state, we depict the same in the above figure for a state s_i; The D1, D2, D1D2 represent the D1R-, D2R-, D1R-D2R MSNs, respectively, and w denotes subscript- corresponding cortico-striatal weights. The schematic also have the representation of DA forms: (1) The δ affecting the cortico-striatal connection weights (Schultz et al., ; Houk et al., 2007), (2) The δ_U affecting the action selection at the GPi (Chakravarthy and Balasubramani, 2014), (3) The Q affecting the D1/D2 MSNs (Schultz, 2010b); and 5HT forms represented by α_D1, α_D2, and α_D1D2 modulating the D1R, D2R, and the D1R-D2R co-expressing neurons, respectively. The inset details the notations used in model section for representing cortico-striatal weights (w) and responses (y) of various kinds of MSNs (D1R expressing, D2R expressing, and D1R-D2R co-expressing) in the striatum, with a sample cortical state size of 4, and maximum number of action choices available for performing selection in every state as 2.

Figure 3

Comparison between the experimental and simulated results for the (A) overall choice (B) Unequal EV (C) Equal EV, under Rapid Tryptophan Depletion (RTD) and Baseline (balanced) condition. Error bars represent the Standard Error (SE) with size “N” = 100 (N = number of simulation instances). The experiment (Expt) and the simulation (Sims) results of any condition are not found to be significantly different. Here the experimental results are adapted from Long et al. (2009).

Figure 4

The mean number of errors in non-switch trials (A) as a function of “α” and outcome trial type; Error bars represent standard errors of the difference as a function of “α” in simulation for size “N” = 100 (N = number of simulation instances) (Sims). (B) Experimental error percentages adapted from Cools et al. (2008). Error bars represent standard errors as a function of drink in experiment (Expt). The results in (B) were reported after the exclusion of the trials from the acquisition stage of each block. (C) The mean number of errors in non-switch trials as a function of condition with experimental (Expt) results adapted from Cools et al. (2008). Error bars represent standard errors either as a function of drink in experiment (or α) in simulation for size “N” = 100, with bal and Trp- representing balanced and tryptophan depleted conditions, respectively. The experiment (Expt) and the simulation (Sims) results of any condition or outcome trial type are not found to be significantly different.

Figure 5

The reward punishment sensitivity obtained by simulated (Sims)- PD and controls model to explain the experiment (Expt) of Bodi et al. (2009). Error bars represent the standard error (SE) with N = 100 (N = number of simulation instances). The Sims matches the Expt value distribution closely, and are not found to be significantly different.