Is the encoding of Reward Prediction Error reliable during development?

Abstract

Reward Prediction Errors (RPEs), defined as the difference between the expected and received outcomes, are integral to reinforcement learning models and play an important role in development and psychopathology. In humans, RPE encoding can be estimated using fMRI recordings, however, a basic measurement property of RPE signals, their test-retest reliability across different time scales, remains an open question. In this paper, we examine the 3-month and 3-year reliability of RPE encoding in youth (mean age at baseline = 10.6 ± 0.3 years), a period of developmental transitions in reward processing. We show that RPE encoding is differentially distributed between the positive values being encoded predominantly in the striatum and negative RPEs primarily encoded in the insula. The encoding of negative RPE values is highly reliable in the right insula, across both the long and the short time intervals. Insula reliability for RPE encoding is the most robust finding, while other regions, such as the striatum, are less consistent. Striatal reliability appeared significant as well once covarying for factors, which were possibly confounding the signal to noise ratio. By contrast, task activation during feedback in the striatum is highly reliable across both time intervals. These results demonstrate the valence-dependent differential encoding of RPE signals between the insula and striatum, and the consistency of RPE signals or lack thereof, during childhood and into adolescence. Characterizing the regions where the RPE signal in BOLD fMRI is a reliable marker is key for estimating reward-processing alterations in longitudinal designs, such as developmental or treatment studies.

Keywords: Adolescence; Development; Prediction Error; Reliability; Reward; fMRI.

Fig. 1.

The task trial time course (adapted from Lahat et al., 2016).

Fig. 2.

Reliability of response time across the two time intervals.

Fig. 3.

(A) Success probability values, exemplified for visit 1, each subject is represented by a single curve (probability per trial is the mean across all preceding trials). (B) RPE values are mostly negative, as demonstrated by the cumulative sum of RPE values (calculated per trial using all preceding trials values); each curve represents a single subject, and all visits are shown.

Fig. 4.

RPE encoding. Voxel-wise group effect maps presenting the parametric modulation values of: (A) Visit 1, corrected for FWE with a threshold of p = 0.005 and a minimal cluster size of 102 voxels; (B) Visit 2, upper panel is corrected similarly and the lower panel is with a non-corrected p = 0.005; (C) Visit 3, using an uncorrected p = 0.005 (upper panel), or an uncorrected p = 0.05 (lower panel). (D) Mean ROI RPE encoding levels, across all voxels per region, averaged across all subject per visit (error bars depict the standard error). The effect size for each ROI (Cohen’s d, peak beta value): right insula: visit 1 (−1.17, −0.09), visit 2 (−2.03, −0.12), visit 3 (−1.24, −0.08); left insula: visit 1 (−1.41, −6.62), visit 2 (−3.11, −0.12), visit 3 (−1.25, −0.083); striatum: visit 1 (1.49, 0.074), visit 2 (1.81, 0.046), visit 3 (1.44, 0.05).

Fig. 5.

Reliability of RPE signals. (A, B) ICC maps (top panels), thresholded for FWE and ICC>0.45 and the respective maps of significant (p < 0.05) ICC z scores (lower panels). (C) Mean RPE encoding over reliable insular voxels across visits, and (D) shows the within-subject comparison of RPE encoding (where each data point is mean RPE signal value, of one subject, in two of the visits).

Fig. 6.

(A) Reliability of general feedback activation, with no RPE modulation, across the three time points. (B) Reliability of only loss trials feedback, transformed to z-score (p = 0.01) across visit 1 to 2 (upper panel) and visit 1 to 3 (lower panel). Under this more stringent threshold only striatum remains reliable. (C) Contrast images of loss feedback reliability versus RPE encoding reliability, visit 1 to 2 and for visit 1 to 3 in panel (D).