Neural integration underlying naturalistic prediction flexibly adapts to varying sensory input rate

Abstract

Prediction of future sensory input based on past sensory information is essential for organisms to effectively adapt their behavior in dynamic environments. Humans successfully predict future stimuli in various natural settings. Yet, it remains elusive how the brain achieves effective prediction despite enormous variations in sensory input rate, which directly affect how fast sensory information can accumulate. We presented participants with acoustic sequences capturing temporal statistical regularities prevalent in nature and investigated neural mechanisms underlying predictive computation using MEG. By parametrically manipulating sequence presentation speed, we tested two hypotheses: neural prediction relies on integrating past sensory information over fixed time periods or fixed amounts of information. We demonstrate that across halved and doubled presentation speeds, predictive information in neural activity stems from integration over fixed amounts of information. Our findings reveal the neural mechanisms enabling humans to robustly predict dynamic stimuli in natural environments despite large sensory input rate variations.

Fig. 1. Hypotheses, stimuli, and paradigm.

a Hypotheses. Neural sensory history integration (SHI) can be limited by a fixed duration (Hypothesis 1, highlighted red, temporal bottleneck) or a fixed amount of information (Hypothesis 2, highlighted blue, informational bottleneck), resulting in a different number of neurally integrated tones ($k’$) across-tone duration conditions. b Full stimulus set. Tone sequences consisted of 34 tones [black squares] ordered by temporal dependence level ($\beta$, rows) and theoretically predicted final tone ($p_{34}^{*}$ [color-coded arrows], columns). For each beta level, we generated three sequences with tone pitch between 220 and 880 Hz. Sequences were chosen to have a $p_{34}^{*}$ lower than 440 Hz (column 1, blue arrows), equal to 440 Hz (column 2, turquoise arrows), or higher than 440 Hz (column 3, yellow arrows). For all sequences, the penultimate tone ($p_{33}$) was 440 Hz. The final presented tone ($p_{34}$ [empty black squares]) was pseudo-randomly drawn from one of six possible tone pitch values at 4, 8, or 12 semitones above or below 440 Hz. Sequences were presented with different tone durations (150 ms, 300 ms, 600 ms per tone). c Trial structure. After stimulus presentation, subjects rated the final tone pitch likelihood given the previous sequence information on a scale of 1–5. Subsequently, subjects rated the trend strength (i.e., beta level) of the presented sequence on a scale of 1–3.

Fig. 2. Subjects’ behavior demonstrates effective final tone pitch prediction (n = 20 participants).

a Final tone pitch likelihood ratings averaged across all tone duration conditions. Final tone pitch likelihood rating (y-axis; 1 = unlikely, 5 = likely) is plotted as a function of presented final tone pitch ($p_{34}$ [empty black squares], x-axis) and theoretically predicted final tone pitch ($p_{34}^{*}$; color-coded). $p_{34}^{*}$ tone pitch is indicated on the x-axis by color-coded arrows. A repeated-measures ANOVA shows a significant interaction between $p_{34}$and $p_{34}^{*}$ [$F$_2,46 = 36.96, $p$ < 0.001, ${\mathrm{\eta }}_p^2$ = 0.66], indicated by the crossover of the blue and yellow lines. Dots represent individual participant data. Data are presented as mean ± SEM across participants. b Final tone pitch likelihood ratings per tone duration condition. Same format as a.

Fig. 3. Slow arrhythmic neuromagnetic activity contains predictive information about upcoming tone pitch (n = 20 participants).

a Prediction analysis schematic and group-level neuromagnetic correlates of theoretically predicted final tone pitch ($p_{34}^{*}$ [color-coded]) prediction for data from the 300 ms tone duration condition. Non-baseline-corrected neuromagnetic activity averaged across 50 ms time windows during the penultimate tone ($p_{33}$) was regressed onto $p_{34}^{*}$ to reveal sensor clusters where neuromagnetic activity is predictive of future tone pitch. Topoplots show $t$-values corresponding to a group-level one-sample $t$-test on regression coefficients for each sensor and time window. White dots indicate significant predictive processing clusters (all $p$ < 0.05, cluster-based permutation test, two-tailed). Right inset shows neuromagnetic activity (unit: T = Tesla) during $p_{33}$-presentation as a function of $p_{34}^{*}$, averaged across sensors for the left (bottom) and right (top) predictive processing cluster defined in the 50–100 ms time window. Data are presented as mean ± SEM across participants. b Event-related fields (ERF) over the course of tone sequence presentation ($p_1$ = tone 1 within the current sequence) for tone sequences with low vs. high $p_{34}^{*}$ (example shown for data from the 300 ms tone duration condition). Top panel: ERF computed for the right predictive processing cluster defined from the 50–100 ms time window (inset and Fig. 3a). Bottom panel: ERF computed for the early sensory filter (inset shows corresponding sensor weights). Gray shading indicates significant differences between trials with low and high $p_{34}^{*}$ (all $p$ < 0.05, cluster-based permutation test, two-tailed). Data are presented as mean across participants.

Fig. 4. Sensory history integration analysis schematic.

I Non-baseline-corrected MEG activity, $N_{s,w}$ for sensor $s$ and time window $w$, in response to the $i$-th tone (16≤ $i$ ≤32) is linearly regressed onto the current tone pitch ($p_i$) and the pitch of $k’$ preceding tones. The $k’$-value providing the best model fit is determined by cross-validation, indicating the number of preceding tone pitches best explaining $N_{s,w}$. II $k’$-values from experimental data (green) are compared against a null distribution constructed by shuffling tone order within each sequence ($k_{shuff}^{\prime }$, gray). III $k’$-values computed for predictive processing clusters are compared across-tone duration conditions within a 3-D coordinate system (axes represent tone duration-specific $k’$). Sensor-wise $k’$values are shown as green dots, $k_{shuff}^{\prime }$-values are shown in gray. Diamonds represent the center of mass for the respective distribution (for visualization only). Blue (informational bottleneck) and red (temporal bottleneck) orientation lines indicate the hypothesis-derived location of $k’$-values. Vector norm (i.e., distance from origin) and angle (i.e., angle to respective orientation lines) for $k’$-values are statistically compared against the shuffled null distribution (histograms). $k’$-values were additionally projected onto a 2-D plane defined by the orientation lines.

Fig. 5. Sensory history integration operates differently across spatial locations, but predictive processing clusters exhibit flexible scaling to sensory input rate (n = 20 participants).

a Across-tone duration-condition comparison of the number of preceding tones best explaining neuromagnetic activity ($k’$-values) from the left and right predictive processing cluster (300 ms tone duration condition, 50–100 ms time window). $k’$-values from experimental data in both sensor clusters (3-D and 2-D plots, green dots) reside significantly (top histograms, $p$ < 0.001, nonparametric permutation test, one-tailed) further away from origin at $k’$ ≈ 6 (i.e., 7 integrated tones) than shuffled data (gray dots). Experimental $k’$-values in the left sensor cluster reside significantly closer to the information line than shuffled data (left middle histogram, $p$ = 0.014, nonparametric permutation test, one-tailed). Additional results from predictive processing clusters defined using other time windows are shown in Supplementary Fig. 5. b A data-driven analysis across the entire sensor array identified a sensor cluster in which SHI behaves according to the informational bottleneck hypothesis. This sensor cluster overlaps with the right predictive processing cluster shown in a. Topoplot shows the angle between experimental data and the information line for all sensors. Significant sensors where the angle is smaller than shuffled data are shown in white ($p$ < 0.05, cluster-based permutation test, one-tailed). Here, $k’$-values reside significantly further away from origin (left histogram, $p$ < 0.001) and significantly closer to the information line than shuffled data (middle histogram, $p$ < 0.001). c A sensor cluster in which SHI behaves according to the temporal bottleneck hypothesis, identified by the data-driven analysis. This sensor cluster has minimal overlap with the predictive processing clusters. Topoplot shows the angle between experimental data and the duration line for all sensors, and significant sensors where the angle is smaller than shuffled data are shown in white ($p$ < 0.05, cluster-based permutation test, one-tailed). Here, $k’$-values reside significantly further away from origin (left histogram, $p$ < 0.001) and significantly closer to the duration line than shuffled data (right histogram, $p$ < 0.001).

Fig. 6. Number of neurally integrated tones correlates with behavioral indices of sensory history dependence across subjects (n = 20 participants).

a Across-subject Spearman correlation of the number of preceding tones best explaining neuromagnetic activity ($k’$-values; averaged across-tone duration conditions and sensors in predictive processing clusters defined from the 300 ms condition) and $F$-statistics of the interaction effect between theoretically predicted final tone pitch ($p_{34}^{*}$) and presented final tone pitch ($p_{34}$; derived from a three-way repeated-measures ANOVA, factors: tone duration, $p_{34}^{*}$, $p_{34}$). In the scatter plots, each dot represents one subject. Red dots represent the seven subjects who received behavioral training, for whom data from the behavioral training session (with identical task paradigm as during the MEG session) were used. Results obtained using behavioral data from the MEG session for all subjects were qualitatively similar. Significant negative correlations (0–50 ms: $p$ = 0.014; 50–100 ms: $p$ = 0.012; all $p$ < 0.05 after FDR correction) were found for right sensor clusters at 0–50 ms and 50–100 ms. b Sensor-wise across-subject Spearman correlation of $k’$-values (averaged across-tone duration conditions for each sensor and TW) and $F$-statistics of the $p_{34}$ × $p_{34}^{*}$ interaction effect (derived from a three-way repeated-measures ANOVA, factors: tone duration, $p_{34}^{*}$, $p_{34}$) computed across the entire sensory array. White dots indicate significant sensor clusters ($p$ = 0.002, cluster-based permutation test, two-tailed).