Worth the Work? Monkeys Discount Rewards by a Subjective Adapting Effort Cost

Abstract

All life must solve how to allocate limited energy resources to maximize benefits from scarce opportunities. Economic theory posits decision makers optimize choice by maximizing the subjective benefit (utility) of reward minus the subjective cost (disutility) of the required effort. While successful in many settings, this model does not fully account for how experience can alter reward-effort trade-offs. Here, we test how well the subtractive model of effort disutility explains the behavior of two male nonhuman primates (Macaca mulatta) in a binary choice task in which reward quantity and physical effort to obtain were varied. Applying random utility modeling to independently estimate reward utility and effort disutility, we show the subtractive effort model better explains out-of-sample choice behavior when compared with parabolic and exponential effort discounting. Furthermore, we demonstrate that effort disutility depends on previous experience of effort: in analogy to work from behavioral labor economics, we develop a model of reference-dependent effort disutility to explain the increased willingness to expend effort following previous experience of effortful options in a session. The result of this analysis suggests that monkeys discount reward by an effort cost that is measured relative to an expected effort learned from previous trials. When this subjective cost of effort, a function of context and experience, is accounted for, trial-by-trial choices can be explained by the subtractive cost model of effort. Therefore, in searching for net utility signals that may underpin effort-based decision-making in the brain, careful measurement of subjective effort costs is an essential first step.SIGNIFICANCE STATEMENT All decision-makers need to consider how much effort they need to expend when evaluating potential options. Economic theories suggest that the optimal way to choose is by cost-benefit analysis of reward against effort. To be able to do this efficiently over many decision contexts, this needs to be done flexibly, with appropriate adaptation to context and experience. Therefore, in aiming to understand how this might be achieved in the brain, it is important to first carefully measure the subjective cost of effort. Here, we show monkeys make reward-effort cost-benefit decisions, subtracting the subjective cost of effort from the subjective value of rewards. Moreover, the subjective cost of effort is dependent on the monkeys' experience of effort in previous trials.

Keywords: adaptation; choice; preference; utility; value.

Figure 1.

Task design. A, Joystick with variable effort. B, Frictional resistance as function of voltage controlling magnet strength. C, Task design. Use of joystick for binary choice between options that differ in effort required. D, Fractal stimuli indicating effort required (in Newton), value bars indicating juice reward amount (in milliliters).

Behavioral and kinematic analysis validation and characterization of task design. A, B, Both animals chose more reward over less reward, given effort being equal. C, D, Monkeys chose less effort over more effort, given equal reward. E, F, Standardized coefficients from logistic regression on choosing left side, including side bias, reward and effort quantity, and bias toward previously chosen side. Column represents overall average, dots represent per-session estimates G, Average profile of position, velocity, and strain for Monkey U at three separate effort levels. Time zero indicates the appearance of the Go cue. H, Average strain in the 100 ms before the Go cue (shaded area) separated by chosen side and effort level. In general, both animals were applying greater strain to the joystick for higher effort levels, reflecting preparation for the greater resistance to movement and therefore demonstrating an understanding of the effort cues. Two-way ANOVA on effort level and chosen direction; chosen effort effect: Monkey U: F_(5,6021) = 17.5, p < 0.001; Monkey W: F_(5,4707) = 24.57, p < 0.001. Error bars are SEM.

Figure 3.

Psychometric construction of common value scale. A, Task design. To establish the equivalent value of effort in units of reward the indifference point between two levels of effort was titrated. Against a fixed option (here, 0.4-ml reward, lowest effort level), a variable option was presented in which the effort level and reward was changed between trials. B, Example psychometric function fit for a single trial block in Monkey W (160 trials). Lines and shaded area represents fit and fit SEM. Value of effort determined from distance on reward-axis at the indifference point (choice p = 0.5 of each option) for each effort level. C, Average equivalent reward value of each effort level for Monkey U (left) and Monkey W (right). In general, higher effort values corresponded to more negative reward values. Effort values were more negative for the smaller monkey (Monkey W), potentially reflecting a different in strength. Red dots are average over all sessions, error bars are SEM, transparent dots are session averages.

Figure 4.

Random utility modeling and results. A, Theory of random utility modeling for reward and effort. Random utility assumes that choice probabilities are determined by the different in utility of (ΔU), not the objective quantity difference (Δx) plus some fixed error. This shows the relationship between three choice sets that have the same objective quantity difference in reward (top) and effort (bottom) but different utility differences and the expected choice probabilities for these six pairs of choices. Random utility model thus uses these observed binary choice probabilities to infer the underlying utility functions. Utility differences are exaggerated for clarity. B, Monotonically increasing utility with reward amount; consistency across four different fitting functions. C, Estimation of reward utility and effort disutility functions from choice options that only differed in reward or in effort. Out-of-sample validation was conducted by predicting choices using the best fitting reward and effort disutility functions to predict choices that were not used to estimate either. D, Near-monotonically decreasing effort (dis)utility with effort (N, Newton). E, Out-of-sample psychometric assessment of utility difference between leftward and rightward choices. F, Quantitative comparisons of fits for the four utility functions tested. G, Model fits for three commonly used combined utility models.

Figure 5.

Juice consumption induces satiety effects. A, With increasing reward consumption (quartiles 1–4), the choice difference between the same options narrowed, possibly reflecting satiety. For each quartile, the left column displays the probability of choosing the option yielding 0.20 ml, whereas the right column shows the choice probability for the option yielding 0.25 ml. Data from Monkey W. Error bars are SEM. B, Utilities for different amounts of consumed reward, as modeled by including total reward consumption as a latent variable in the reward utility model, thus quantifying the reduced reward utility as reward consumption increased. The four lines refer to the four quartiles (top to bottom).

Figure 6.

Effort choice history influences sensitivity to effort. A, For each quartile of daily trials, the left column displays the probability of choosing the low effort option (0 N), whereas the right column shows the choice probability for the high effort option (8 N) in Monkey W. B, Quadratic fits of effort disutility for the four quartiles suggest effort disutility decreases with total effort expenditure over each session. The four curves refer to the four quartiles (top to bottom). Error bars show standard errors of the parameter estimate.

Figure 7.

Reference-dependent effort preferences. A, Piecewise-linear effort model. Effort disutility is the difference between effort and the effort reference point, scaled by a slope constant that is dependent on whether the effort is above or below the reference point. B, Reference point models. The effort reference point was either modeled as the average of the previous 30 efforts (moving average model) or learnt from previous efforts in with a learning rate dependent on whether the effort was above or below the reference point (asymmetric Rescorla–Wagner model). C, Effort disutility for Monkeys U and W, as scaled to the same utility as reward utility in Figure 4. Reference was modeled as a moving average of previous chosen effort. D, Reference point model from Rescorla–Wagner reinforcement learning model. In all cases, the slope above the reference point was significantly greater than below the reference point, suggesting reference-dependent preferences. Shaded ribbon is 1 SD of model fit.