Risk-sensitive optimal feedback control accounts for sensorimotor behavior under uncertainty

Abstract

Many aspects of human motor behavior can be understood using optimality principles such as optimal feedback control. However, these proposed optimal control models are risk-neutral; that is, they are indifferent to the variability of the movement cost. Here, we propose the use of a risk-sensitive optimal controller that incorporates movement cost variance either as an added cost (risk-averse controller) or as an added value (risk-seeking controller) to model human motor behavior in the face of uncertainty. We use a sensorimotor task to test the hypothesis that subjects are risk-sensitive. Subjects controlled a virtual ball undergoing Brownian motion towards a target. Subjects were required to minimize an explicit cost, in points, that was a combination of the final positional error of the ball and the integrated control cost. By testing subjects on different levels of Brownian motion noise and relative weighting of the position and control cost, we could distinguish between risk-sensitive and risk-neutral control. We show that subjects change their movement strategy pessimistically in the face of increased uncertainty in accord with the predictions of a risk-averse optimal controller. Our results suggest that risk-sensitivity is a fundamental attribute that needs to be incorporated into optimal feedback control models.

Figure 1. Schematic of the task.

Subjects attempted to move a virtual ball (represented by the green circle) to the center of a target line (represented by the black horizontal line). The ball moved with constant y-velocity and hit the target after 1 s, whereas it moved with Brownian motion in the x-direction. Final positional errors were penalized by a quadratic cost function that was displayed as a parabola and the error cost was displayed at the end of the trial (blue bar). Subjects could exert control on the x position of the ball by moving their hand to the left or right (gray solid and dashed arrow lines). This incurred a control cost which was the quadratic in the control signal and the cumulative across a trial (yellow bar) was constantly displayed. At the end of the trial subjects received feedback of the total cost, the sum of control and error cost (yellow-blue bar). Subjects were required to minimize the total cost on average and were tested on four conditions (2 noise levels×2 control cost levels). The path taken by the ball is shown for a typical trial.

Figure 2. Task performance.

A. All 250 ball paths for a typical subject for the low noise level (high control cost condition). Individual trials are colored randomly. B. as A. but for the high noise level. C. Mean control magnitude across all trials and subjects for the low noise level (blue - low cost condition, green - high cost condition). D. as C. but for the high noise level (yellow - low cost condition, red - high cost condition). E. Mean absolute positional error (absolute deviation from the center of the target line) across all trials and subjects for the low noise level (colors as in C.) The dashed line shows the mean absolute error if subjects did not intervene. F. as E. but for the high noise level. Note that the y-scale in D. and F. is five times greater than in C. and E. due to the higher noise level. Shaded area shows one s.e.m. across all trials.

Figure 3. Predictions of optimal feedback control models.

A risk-neutral optimal control model , attempts to minimize the mean of the cost function. As a result, its policy (that is the motor command applied for a given state of the world) is independent of the noise variance N. In contrast, a risk-sensitive optimal control model , minimizes a weighted combination of the mean and variance of the cost. Additional variance is an added cost for a risk-averse controller (), whereas it makes a movement strategy more desirable for a risk-seeking controller (). As a consequence, the policy of the controller changes with the noise level N depending on its risk-attitude . A.–C. Changes in motor command with the state of the ball (its positional deviation from the center) for a low noise level (green) and for a high noise level (red) for the risk-neutral (A), risk-averse (B) and risk-seeking (C) controllers. The slope of the lines is equivalent to the control gain of the controller. D.–F. Contribution of control cost to total cost (control cost+error cost) for the risk-neutral (D), risk-averse (E) and risk-seeking (F) controllers.

Figure 4. Risk-sensitivity.

A. Results of the multilinear regression analysis of the low control cost conditions for subject number 5. The line shows the average motor command that the subject produces for a given position (blue - low noise level, yellow - high noise level). The slope of the line is a measure for the position gain of the subject. B. same as in A. but for the high control cost conditions (green - low noise level, red - high noise level). C.–F. Compares various measures between the high and low noise conditions. A risk-neutral controller predicts values to be the same for both condition (dashed line), a risk-averse controller predicts values to fall above the dashed line and a risk-seeking controller below it. C. Negative position gain for the high noise condition plotted against the low noise condition for all six subjects in the low control cost conditions (subject 5 in black, ellipses show the standard deviation). The dashed line represent equality between the gains. D. as C. but for the high control cost conditions. E. Negative velocity gain for the high noise condition plotted against the low noise condition for all six subjects for the low control cost conditions (ellipses show the standard deviation). F. as E. but for the high control cost conditions.

Figure 5. Difference in position and velocity gains.

A. The difference in velocity gains plotted against the difference in position gains of all subjects for the low control cost conditions (ellipses show the 95% confidence region). The color gradient indicates the values predicted by the simulations of a risk-sensitive optimal controller for different -values. B. as A. but for the high control cost conditions. C. Subjects' individual risk-parameters inferred from the experimental data of the high cost level versus inferred from the data of the low cost level (ellipses show 1 s.d.).

Figure 6. Analysis of possible confounds and sensorimotor delay.

A. Results of the multilinear regression analysis of the first 15 trials for the low and the high noise condition. The subjects' data was pooled according to whether they began the experiment with a low gain or with a high gain (green and blue - low gain first; red and yellow - high gain first). B. Number of trials for different angles of the velocity vector of the ball with the wall upon impact (dark red - hypothetical impact angle of the ball had the subjects not intervened, dark blue - actual impact angle during the experiment). C. R-values of the multilinear regression analysis averaged across all subjects and conditions for different sensorimotor delays.

Figure 7. Simulations of an optimal controller with incomplete state observation and sensorimotor delay.

A.–C. Changes in motor command with position for a fixed velocity () for the low noise level (green) and for the high noise level (red). D.–F. Contribution of control cost to total cost (control cost+error cost). A. & D. - Predictions of a risk-neutral controller. B. & E. - Predictions of a risk-averse controller. C. & F. - Predictions of a risk-preferring controller.

Figure 8. Contribution of control cost to total cost, and extra cost from risk sensitivity.

A. Contribution of control cost to total cost for the high noise condition plotted against the low noise condition for the low control cost conditions (ellipses show 1 s.e.m. across all 250 trials). B. as A. but for the high cost level. C. Estimated extra cost in percent of a risk-sensitive controller with incomplete observation and sensorimotor delay based on the experimentally inferred -parameters for the low cost level. D. as C. but for the high cost level. E. Relationship between extra cost of a risk-sensitive controller relative to a risk-neutral controller for a range of -values overlaid with the subjects' experimentally inferred -parameters standard deviation for the low cost level. F. as E. but for the high cost level.