A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research

Abstract

Foraging is a fundamental behavior, and many types of animals appear to have solved foraging problems using a shared set of mechanisms. Perhaps the most common foraging problem is the choice between exploiting a familiar option for a known reward and exploring unfamiliar options for unknown rewards-the so-called explore/exploit trade-off. This trade-off has been studied extensively in behavioral ecology and computational neuroscience, but is relatively new to the field of psychiatry. Explore/exploit paradigms can offer psychiatry research a new approach to studying motivation, outcome valuation, and effort-related processes, which are disrupted in many mental and emotional disorders. In addition, the explore/exploit trade-off encompasses elements of risk-taking and impulsivity-common behaviors in psychiatric disorders-and provides a novel framework for understanding these behaviors within an ecological context. Here we explain relevant concepts and some common paradigms used to measure explore/exploit decisions in the laboratory, review clinically relevant research on the neurobiology and neuroanatomy of explore/exploit decision making, and discuss how computational psychiatry can benefit from foraging theory.

Figure 1

Conceptual overview of the explore/exploit trade-off. Decisions may vary along a continuum between exploration and exploitation with the most advantageous behaviors occurring around a point of balance between the two. Around this balance, there may be slight externally or internally driven biases towards one decision vs the other. Extreme exploitative or exploratory decisions that are not adaptations to the environment may be disadvantageous, leading to either inefficiency and lack of expertise (ie, overly exploratory) or habit formation and motivational deficiencies (ie, overly exploitative).

Figure 2

Design of the 4-armed bandit task. (a) In each trial, four options (ie, slot machines) are presented. The player selects one option, then the rewards (ie, points) paid off for that option on that trial are shown. (b) Example of the latent value structure of the options across trials. The value of each option changes gradually and independently of the other options. Based on the decision rule described in Daw et al (2006), exploratory choices made by a player are marked with closed circles and exploitative choices are marked with open squares.