Optimal inference with suboptimal models: addiction and active Bayesian inference

Abstract

When casting behaviour as active (Bayesian) inference, optimal inference is defined with respect to an agent's beliefs - based on its generative model of the world. This contrasts with normative accounts of choice behaviour, in which optimal actions are considered in relation to the true structure of the environment - as opposed to the agent's beliefs about worldly states (or the task). This distinction shifts an understanding of suboptimal or pathological behaviour away from aberrant inference as such, to understanding the prior beliefs of a subject that cause them to behave less 'optimally' than our prior beliefs suggest they should behave. Put simply, suboptimal or pathological behaviour does not speak against understanding behaviour in terms of (Bayes optimal) inference, but rather calls for a more refined understanding of the subject's generative model upon which their (optimal) Bayesian inference is based. Here, we discuss this fundamental distinction and its implications for understanding optimality, bounded rationality and pathological (choice) behaviour. We illustrate our argument using addictive choice behaviour in a recently described 'limited offer' task. Our simulations of pathological choices and addictive behaviour also generate some clear hypotheses, which we hope to pursue in ongoing empirical work.

Fig. 1

Task for model validation. (A) Each game comprised a specified number of trials (discrete time-steps). On each trial, subjects had to decide whether to accept a current offer or wait for the next trial. If subjects decided to wait, the low offer could be withdrawn with probability r, it could be replaced by a high offer with probability q or could be carried over to the next trial with probability 1 − (r + q). (B) The probabilities were defined by hazard rates, such that withdrawal probability increased, whereas a high offer became less likely the longer subjects waited. Adapted from .

Fig. 2

Results. Observed and estimated acceptance probabilities as predicted by our model (note that P_wait = 1 − P_accept). These estimates are based upon maximum a posteriori values for prior precision, the hazard rate and sensitivity to differences in monetary cues.

Fig. 3

Proposed variational message passing scheme. This figure illustrates the proposed message passing scheme in active inference and its putative mapping to brain function. The scheme rests on variational updates, such that several distinct representations (expectations of states, polices and precision) are updated and inform each other – providing a nice metaphor for functional segregation and integration in the brain. Here, current observations inform inference about hidden states of the environment, which induce updates of precision. Precision influences policy selection which in turn specifies actions – that are sampled from the most likely policies. Note that the connections are recursive because variation updates require a reciprocal exchange of sufficient statistics. Recent empirical work has shown that expected precision is encoded in the dopaminergic midbrain .

Fig. 4

Effects of precision on behaviour. (A–C) reflect low (α = 4), medium (α = 8) and high (α = 16) prior precision, respectively. The left column shows expectations about latent states and precision in an example game with initial offer withdrawal at trial three. In each panel, the top row shows the inferred hidden state the agent believes this is in (upper left) and the expected probability of accepting or declining as a function of trials (upper right panel: dotted lines for earlier trials and the solid lines for the last trial). These expectations are updated using expected precision that is shown as a function of trial in the lower row – in terms of the precision at the end of each trial (lower left) and simulated dopamine responses reflecting the changes in precision (lower right). See for further details about these (variational) updates. Here, the initial propensity to wait (stay) depends on prior precision and increases for higher values of precision (upper right panel). Intuitively, the drop of expected precision when the initial offer is withdrawn is larger with higher prior precision. The middle column shows the equivalent updates when a high offer is presented in the fifth trial. Expected precision displays a sharp increase when the high offer is received, where the size of the increase again depends on individual prior precision. The right column shows the observed acceptance probabilities at each time step, simulated in 256 games. It becomes obvious that the acceptance latencies are shifted towards later trials with increasing precision (confidence that the high offer will be received).

Fig. 5

Effects of estimates of the hazard rate on behaviour. (A–C) reflect high (losing the initial offer is likely), medium and low (losing the initial offer is unlikely) estimates of the hazard rate, respectively. This figure uses the same format as Fig. 4: the left and middle column show expectations about hidden states, behaviour and expected precision for a withdrawal of the initial offer (at the third trial) and the receipt of a high offer at the fifth trial, respectively. This time, the behaviour and initial level of precision is identical, but the expected behaviour reflects the differences in hazard rates – with the probability of waiting being smaller if losing the initial offer is likely. The right column predicts that subjects will accept the initial offer early if a loss is thought to be likely.

Fig. 6

Effects of a low prior precision and a low estimate of the hazard rate on behaviour. Low precision causes subjects to be more stochastic and less patient in their behaviour, whereas a low estimate of the hazard rate causes subjects to become more risk-seeking in waiting for the high offer. See legends for Figs. 4 and 5 for details.