Goals and the Structure of Experience

Abstract

Purposeful behavior is a hallmark of natural and artificial intelligence. Its acquisition is often believed to rely on world models, comprising both descriptive (what is) and prescriptive (what is desirable) aspects that identify and evaluate state of affairs in the world, respectively. Canonical computational accounts of purposeful behavior, such as reinforcement learning, posit distinct components of a world model comprising a state representation (descriptive aspect) and a reward function (prescriptive aspect). However, an alternative possibility, which has not yet been computationally formulated, is that these two aspects instead co-emerge interdependently from an agent's goal. Here, we describe a computational framework of goal-directed state representation in cognitive agents, in which the descriptive and prescriptive aspects of a world model co-emerge from agent-environment interaction sequences, or experiences. Drawing on Buddhist epistemology, we introduce a construct of goal-directed, or telic, states, defined as classes of goal-equivalent experience distributions. Telic states provide a parsimonious account of goal-directed learning in terms of the statistical divergence between behavioral policies and desirable experience features. We review empirical and theoretical literature supporting this novel perspective and discuss its potential to provide a unified account of behavioral, phenomenological and neural dimensions of purposeful behaviors across diverse substrates.

Keywords: goals; state representation; world models.

Figure 1:. Telic states are sensitive to the goal

(Left) all six possible routes, or experiences, are equally desirable with respect to the goal of reaching the park and can therefore be clustered into a single telic state $S_0$. (Right) for the goal of getting a cup of coffee on the way to the park, these experiences are split into two telic states: $S_1$, containing the routes that do not pass by the local cafe, and $S_2$ containing those that do.

Figure 2:. Agentic vs. telic models of learning:

(Left) traditional models view agents as the fundamental learning units, with dedicated neural modules for state and value estimation. (Right) states are typically assumed to be estimated on the basis of the previous state and the current observation while evaluation relies on stored values for different states which are updated based on a privileged set of observations that are considered “rewarding”. In the telic states framework, purposeful behavior stems from goals, which induce telic states, encapsulating descriptive and evaluative aspects of experience. Goals are represented across multiple brain regions and may even extend beyond the brain, e.g., in cases of multiple agents pursuing a joint goal

Figure 3:. Dual goal navigation task:

each tile shows 500 one-dimensional random walk trajectories of length $T=30$, generated by a Gaussian policy parameterized by the mean $(\mu )$ and standard deviation $(\sigma )$ of position update step (x-axis) across time (y-axis). Regions of interest $R$ and $L$ consist of line segments centered around $x_R=2$ and $x_L=-2$, shown as green and red lines respectively at $T=30$. Trajectories reaching one of the goals are plotted in the corresponding color, illustrating the relationship between policy parameters and goal reaching likelihoods. The default policy, $\left({\mu }_0,{\sigma }_0\right)=(0,1)$, shown in the center gray tile, is equally likely to reach $R$ and $L$.

Figure 4:. Telic state representation learning for navigation task with shifting goals:

points in $(\mu ,\sigma )$ policy space colored by the difference between their probability of reaching unit length regions, $R$ and $L$, centered around 2 and −2 respectively, at time $T=30$. Top left: telic states $S_L$ and $S_R$ (outlined by red and green dashed lines, respectively) consist of policies that are more likely to reach the corresponding region by a threshold of $ϵ=0.1$ or more. Contour lines indicate isometric policy complexity levels, relative to the default policy ${\pi }_0:\left({\mu }_0=0,{\sigma }_0=1\right)$ (black dot), for a capacity bound of $\delta =1$ bit. Green and red dots show the information projection of ${\pi }_0$ on $S_R$ and $S_L$ respectively, i.e., the policies each telic state closest to ${\pi }_0$ in KL-divergence Top right: shifting the center of $R$ to 2.5, renders $S_R$ unreachable from ${\pi }_0$ with $\delta$ bounded policy complexity. The policy ${\pi }_M:\left({\mu }_M,{\sigma }_M\right)$ (yellow dot) is the one closest to $S_R$ while still within the complexity capacity of the agent. Bottom left: splitting $S_R$ by inserting an intermediate telic-state, $S_M$, centered around ${\mu }_M$. By construction, the nearest distribution to ${\pi }_0$ in $S_M$, in the KL sense (orange dot), is within the agent’s complexity capacity. Bottom right: both $S_M$ and $S_R$ are reachable with respect to the agent’s new default policy, ${\pi }_M\left({\mu }_M=1.37,{\sigma }_M=1.15\right)$ (see algorithm 1 for details); the new telic state representation $\left\{S_0,S_L,S_M,S_R\right\}$ is telic controllable with respect to ${\pi }_0(0,1)$, $\delta =1$, and $N=1$.

Figure 5:. Complexity-granularity curves:

Each line shows the policy complexity capacity, relative to the default policy ${\pi }_0(0,1)$ (ordinate) required to reach the corresponding telic state at a given representational granularity level, quantified by the negative log of the sensitivity parameter $ϵ$ (abscissa). Dashed gray lines show the values used in the dual-goal navigation example: $\delta =1$ (horizontal) and $ϵ=0.1$ (vertical)

Figure 6:. Goal-complexity tradeoff curves:

the probability of reaching each telic state as a function of policy complexity. Left: an agent with a default policy ${\pi }_0:\left({\mu }_0,{\sigma }_0\right)=(0,1)$ is unable to reach telic state $S_R$ with a complexity capacity limit of $\delta =1$ (gray vertical line). Right: with ${\pi }_1:\left({\mu }_1,{\sigma }_1\right)=(1.09,1.24)$ as its default policy, the agent can reach both $S_M$ and $S_R$ with the same policy complexity capacity.