The rational use of causal inference to guide reinforcement learning strengthens with age

Abstract

Beliefs about the controllability of positive or negative events in the environment can shape learning throughout the lifespan. Previous research has shown that adults' learning is modulated by beliefs about the causal structure of the environment such that they update their value estimates to a lesser extent when the outcomes can be attributed to hidden causes. This study examined whether external causes similarly influenced outcome attributions and learning across development. Ninety participants, ages 7 to 25 years, completed a reinforcement learning task in which they chose between two options with fixed reward probabilities. Choices were made in three distinct environments in which different hidden agents occasionally intervened to generate positive, negative, or random outcomes. Participants' beliefs about hidden-agent intervention aligned with the true probabilities of the positive, negative, or random outcome manipulation in each of the three environments. Computational modeling of the learning data revealed that while the choices made by both adults (ages 18-25) and adolescents (ages 13-17) were best fit by Bayesian reinforcement learning models that incorporate beliefs about hidden-agent intervention, those of children (ages 7-12) were best fit by a one learning rate model that updates value estimates based on choice outcomes alone. Together, these results suggest that while children demonstrate explicit awareness of the causal structure of the task environment, they do not implicitly use beliefs about the causal structure of the environment to guide reinforcement learning in the same manner as adolescents and adults.

Keywords: Decision making; Human behaviour.

Fig. 1. Experimental design.

Participants were told that they were mining for gold in three territories visited by three different agents who intervened to generate negative (Robber), positive (Millionaire), or random (Sheriff) outcomes. On each trial, participants selected a mine (Choice), observed the outcome associated with their choice (Feedback), and indicated their belief about hidden agent intervention (Attribution). Elements of this image were designed by Freepik (https://www.freepik.com/) and are licensed for personal use.

Fig. 2. Attribution data.

Participant’s beliefs about hidden agent intervention aligned with the experimental manipulation, and attribution rates were higher overall in younger individuals. We also observed an age by reward outcome interaction, indicating that younger individuals attributed positive outcomes to the hidden agent more so than older individuals.

Fig. 3. Observed and simulated learning data.

Across territories, older participants learned to select the best mine faster. Younger participants showed better learning in the Millionaire territory (a). Simulated data using subjects’ fitted parameter estimates for each of the best fitting models are depicted (b–d). Boxes denote the age group that was best fit by the model. Participants are separated by age group (Children: 7–12, Adolescents: 13–17, Adult: 18–25) and trial bins for visualization purposes. The corresponding statistical analyses on the empirical data treat age and trial as continuous variables.

Fig. 4. Computational model comparison.

Children were best fit by a one learning rate model, adolescents were best fit by the adaptive Bayesian model, and adults were best fit by the empirical Bayesian model, as indexed by protected exceedance probabilities (PXPs).