Neurocomputational models of basal ganglia function in learning, memory and choice

Abstract

The basal ganglia (BG) are critical for the coordination of several motor, cognitive, and emotional functions and become dysfunctional in several pathological states ranging from Parkinson's disease to Schizophrenia. Here we review principles developed within a neurocomputational framework of BG and related circuitry which provide insights into their functional roles in behavior. We focus on two classes of models: those that incorporate aspects of biological realism and constrained by functional principles, and more abstract mathematical models focusing on the higher level computational goals of the BG. While the former are arguably more "realistic", the latter have a complementary advantage in being able to describe functional principles of how the system works in a relatively simple set of equations, but are less suited to making specific hypotheses about the roles of specific nuclei and neurophysiological processes. We review the basic architecture and assumptions of these models, their relevance to our understanding of the neurobiological and cognitive functions of the BG, and provide an update on the potential roles of biological details not explicitly incorporated in existing models. Empirical studies ranging from those in transgenic mice to dopaminergic manipulation, deep brain stimulation, and genetics in humans largely support model predictions and provide the basis for further refinement. Finally, we discuss possible future directions and possible ways to integrate different types of models.

Figure 1

Left. Functional anatomy of the basal ganglia circuit, showing an updated model of the primary projections. In addition to the classic “direct” and “indirect” pathways from Striatum to BG output nuclei originating in striatonigral (Go) and striatopallidal (NoGo) cells respectively, the revised architecture features focused projections from NoGo units to GPe and strong top-down projections from cortex to thalamus. Further, the STN is incorporated as part of a newly discovered hyperdirect pathway (rather than part of the indirect pathway as originally conceived), receiving inputs from frontal cortex and projecting directly to both GPe and GPi. Right. Neural network model of this circuit, with four different responses represented by four columns of motor units, four columns each of Go and NoGo units within Striatum, and corresponding columns within GPi, GPe and Thalamus. Fast spiking GABA-ergic interneurons (γ-IN) regulate Striatal activity via inhibitory projections. For implementational details, see Frank (2005, .

Figure 2

a) Probabilistic selection reinforcement learning task. During training, participants select among each stimulus pair. Probabilities of receiving positive/negative feedback for each stimulus are indicated in parentheses. In the test phase, all combinations of stimuli are presented without feedback. “Go learning” is indexed by reliable choice of the most positive stimulus A in these novel pairs, whereas “NoGo learning” is indexed by reliable avoidance of the most negative stimulus B.b) Striatal Go and NoGo activation states when presented with input stimuli A and B respectively. Simulated Parkinson's (Sim PD) was implemented by reducing striatal DA levels, whereas medication (Sim DA Meds) was simulated by increasing DA levels and partially shunting the effects of DA dips during negative feedback. c) Behavioral findings in PD patients on/off medication supporting model predictions (Frank et al., 2004). d) Replication in another group of patients, where here the most prominent effects were observed in the NoGo learning condition (Frank et al., 2007b). e) Similar results in healthy participants on dopamine agonists and antagonists modulating presynaptic DA (pDA) and f) adult ADHD participants on and off stimulant medications. g), h) Individual differences in Go/NoGo learning in college students are predicted by genes controlling striatal D1/D2 function.

Figure 3

a) Subthalamic nucleus contributions to model performance in the probabilistic selection task. While not differing from intact networks in selection among trained low-conflict discriminations (80 vs 20 and 70 vs 30), STN lesioned networks were selectively impaired at the high conflict selection of an 80% positively reinforced response when it competed with a 70% response. The model STN Global NoGo signal prevents premature responding when multiple responses are potentially rewarding, increasing the likelihood of accurate choice (Frank, 2006). b) Behavioral results in Parkinson's patients on and off DBS, confirming model predictions. Response time differences are shown for high relative to low conflict test trials. Whereas healthy controls, patients on/off medication (not shown) and patients off DBS adaptively slow decision times in high relative to low conflict test trials, patients on DBS respond impulsively faster in these trials (adapted from (Frank et al., 2007b)).

Figure 4

Abstract reinforcement learning models can be useful for investigating individual differences. Here a model was used to estimate the impact of reinforcement (winning money or not in a gambling task) on the likelihood of making a low- or high-risk gamble in the subsequent trial. The best-fitting parameter for each subject determines the magnitude and sign of the weight change for the high-risk option after obtaining a high-risk reward. Individual differences in this parameter were then correlated with reinforcement-related brain activation. Results indicate that, in a network of regions including the lateral striatum (top right), this weight-update parameter (x-axis) predicts whether brain activations to large rewards are associated with subsequent risky (y-axis positive values) or non-risky (negative values) choices. In this case, individual differences proved critical for understanding how reinforcements guide subsequent decisions: for some subjects reward-related activity predicted increased likelihood of making a subsequent risky choice, whereas for others it predicted decreased likelihood, according to their estimated parameters. See Cohen and Ranganath, 2005, for details.