Action selection in multi-effector decision making

Abstract

Decision making and reinforcement learning over movements suffer from the curse of dimensionality: the space of possible movements is too vast to search or even represent in its entirety. When actions involve only a single effector, this problem can be ameliorated by considering that effector separately; accordingly, the brain's sensorimotor systems can subdivide choice by representing values and actions separately for each effector. However, for many actions, such as playing the piano, the value of an action by an effector (e.g., a hand) depends inseparably on the actions of other effectors. By definition, the values of such coordinated multi-effector actions cannot be represented by effector-specific action values, such as those that have been most extensively investigated in parietal and premotor regions. For such actions, one possible solution is to choose according to more abstract valuations over different goods or options, which can then be mapped onto the necessary motor actions. Such an approach separates the learning and decision problem, which will often be lower-dimensional than the space of possible movements, from the multi-effector movement planning problem. The ventromedial prefrontal cortex (vmPFC) is thought to contain goods-based value signals, so we hypothesized that this region might preferentially drive multi-effector action selection. To examine how the brain solves this problem, we used fMRI to compare patterns of BOLD activity in humans during reward learning tasks in which options were selected through either unimanual or bimanual actions, and in which the response requirements in the latter condition inseparably coupled valuation across both hands. We found value signals in the bilateral medial motor cortex and vmPFC, and consistent with previous studies, the medial motor value signals contained contra-lateral biases indicating effector-specificity, while the vmPFC value signals did not exhibit detectable effector specificity. Although neither region's value signaling differed significantly between bimanual and unimanual conditions, the vmPFC value region showed greater connectivity with the medial motor cortex during bimanual than during unimanual choices. The specific region implicated, the anterior mid-cingulate cortex, is thought to act as a hub that links subjective value signals to motor control centers. These results are consistent with the idea that while valuation for unilateral actions may be subserved by an effector-specific network, complex multi-effector actions preferentially implicate communication between motor regions and prefrontal regions, which may reflect increased top-down input into motor regions to guide action selection.

Figure 1

Experimental task. (A) The sequence of events during a task trial. Subjects chose a shape within 1.5 seconds of shape onset, whereupon the unchosen shapes dimmed and, after a fixed ISI, subjects saw either a dollar coin indicating they had won money or a scrambled dollar coin indicating they had not. After a variable ITI the next trial began, with the shapes appearing in different positions. The amount of time between when a choice was made and the time limit was added to ITI to ensure consistent timing. (B) The two experimental conditions, bimanual and unimanual. Both conditions required that participants press buttons with their middle and index fingers on each hand to select shapes. A button, or combination of buttons, corresponded to a particular location on the screen and selected whatever shape occupied that location on that trial. The correspondence between buttons and choices is shown above for both conditions, with the sets of four yellow icons representing the four available buttons. The filled-in icons indicate which buttons would be used to select the neighboring shape.

Figure 2

Motor-related activity. (A) Regions of the bilateral precentral gyrus and medial motor cortex showing an effect of right versus left-handed actions. (B) The medial motor cortex ROI (SMA and MCC voxels showing activity across all motor conditions; see methods) showed greater activation during bimanual than either type of unimanual movement in the right hemisphere, with a similar trend in the left hemisphere.

Figure 3

Correlates of value and lateralization thereof. We found correlates of chosen value in the (A left) SMA and MCC and in the (B left) ventral-medial prefrontal cortex, as well as (C left) RPE signals in the ventral striatum. The medial motor activation contained two distinct pairs of contralateral peaks, so we divided the activation into four ROIs and tested for lateralization between contralateral ROIs. At our default threshold only the right vmPFC contained significant voxels, so in order to perform a similar test of value laterality we relaxed the threshold to p < 0.01 (voxels active only at the lower threshold are displayed as semi-transparent). Examining both chosen value and action value effect sizes in these ROIs (action values shown), we find in the anterior SMA/MCC a significant hemisphere×effector interaction (A right), but not in the vmPFC (B right). We also find an effector×hemisphere interaction in the striatal RPE responses (C right).

Figure 4

Functional connectivity with the vmPFC. (A) Voxels in the SMA/MCC showing higher vmPFC connectivity during bimanual choices than unimanual choices overall, collapsing across right and left-handed choices. (B) The conjunction of bimanual connectivity greater than right and bimanual greater than left-hand action connectivity, verifying that the effects are not driven by asymmetries between right and left action connectivity strengths. While we find activity in both the MCC and SMA, only the MCC survives small-volume cluster correction.