Estimating properties of the fast and slow adaptive processes during sensorimotor adaptation

Abstract

Experience of a prediction error recruits multiple motor learning processes, some that learn strongly from error but have weak retention and some that learn weakly from error but exhibit strong retention. These processes are not generally observable but are inferred from their collective influence on behavior. Is there a robust way to uncover the hidden processes? A standard approach is to consider a state space model where the hidden states change following experience of error and then fit the model to the measured data by minimizing the squared error between measurement and model prediction. We found that this least-squares algorithm (LMSE) often yielded unrealistic predictions about the hidden states, possibly because of its neglect of the stochastic nature of error-based learning. We found that behavioral data during adaptation was better explained by a system in which both error-based learning and movement production were stochastic processes. To uncover the hidden states of learning, we developed a generalized expectation maximization (EM) algorithm. In simulation, we found that although LMSE tracked the measured data marginally better than EM, EM was far more accurate in unmasking the time courses and properties of the hidden states of learning. In a power analysis designed to measure the effect of an intervention on sensorimotor learning, EM significantly reduced the number of subjects that were required for effective hypothesis testing. In summary, we developed a new approach for analysis of data in sensorimotor experiments. The new algorithm improved the ability to uncover the multiple processes that contribute to learning from error. NEW & NOTEWORTHY Motor learning is supported by multiple adaptive processes, each with distinct error sensitivity and forgetting rates. We developed a generalized expectation maximization algorithm that uncovers these hidden processes in the context of modern sensorimotor learning experiments that include error-clamp trials and set breaks. The resulting toolbox may improve the ability to identify the properties of these hidden processes and reduce the number of subjects needed to test the effectiveness of interventions on sensorimotor learning.

Keywords: expectation maximization; motor learning; two-state model.

Fig. 1.

Experimental paradigm: expectation maximization (EM) and least mean square error estimation (LMSE) algorithms uncover different hidden processes. A: subjects (n = 20) participated in a reach adaptation task. There were 8 targets in total, each chosen pseudo-randomly and presented once in epochs of 8 trials. Following a no-perturbation baseline period, a 30° counterclockwise rotation was applied to the cursor representing the subject’s hand position. After 30 epochs of this perturbation, visual feedback was removed for 15 epochs. Finally, visual feedback was reinstated during a washout block of 30 epochs. B: single-subject behavior. We fit the epoch-by-epoch data (reach direction) of each subject with EM (blue lines; top) and LMSE (red lines; top). Both provide good fits to the measured data. Each algorithm estimated the fast and slow processes that produced the measured behavior (bottom). For subject S9, these time courses agreed across algorithms. For subject S16, EM and LMSE time courses exhibited reasonable two-state behavior but had differing learning dynamics. For subjects S2 and S1, the EM and LMSE predictions diverged completely.

Fig. 2.

Comparison of parameter values uncovered by EM and LMSE as fitted to experimental data. A: population behavior, represented by the average time course across all 20 subjects. Top: the average behavior (black) is shown overlaid with the average EM (blue) and average LMSE (red) fits. EM and LMSE had very similar fits to the behavior. However, the algorithms’ predictions regarding the slow and fast states diverged. Error bars indicate ± 1 SE. B: model parameters. Bars indicate the mean value across the subjects. Error bars indicate ± 1 SE. C: we compared the corrected Akaike Information Criterion (AIC) of 2 competing likelihood models: one with state and motor noise, and one without state and motor noise. AICc was lower for a model with state and motor noise for 14 of the 20 subjects (black lines) and larger for 6 of the 20 subjects (gray lines). A paired t-test across subjects indicated that a model with motor and state noise possessed a lower AICc (that is, a better fit) than a model without these noise sources.

Fig. 3.

Simulated paradigms and behavior. A: we simulated two-state models of learning in the context of 4 behavioral paradigms with abrupt and gradual introduction of perturbations and with error clamp (EC) trials and set breaks. B: the expected value of the measured behavior (black), fast state (green), and slow state (blue) of learning. These time courses correspond to two-state model parameters extracted from our subject population (Table 1). C: for each of the 4 paradigms, behavior was simulated according to a two-state model of learning (see Eq. 10); 1,000 simulations were performed for each paradigm. The two-state model parameters were fixed for each simulation, solely the seed for the random number generator varied from simulation to simulation. Here, we provide an example of a behavioral trajectory for each of the 4 paradigms. We fit each trajectory using EM and LMSE. D: the true slow state of learning along with EM and LMSE predictions. In the example simulations of paradigms 1–3, LMSE failed to capture the slow state of learning. In paradigm 4, both LMSE and EM closely tracked the true slow state. E: the true fast state of learning along with EM and LMSE predictions. For paradigms 1–3, LMSE predictions diverged from the true fast state trajectory. For paradigm 4, both EM and LMSE tracked the true fast-state time course.

Fig. 4.

Performance of EM and LMSE algorithms. For each paradigm, 1,000 simulations were performed with fixed two-state model parameters but a varying seed for the random number generator, altering noise. EM and LMSE were used to fit a two-state model to the simulated behavior. The EM and LMSE parameters were used to simulate noise-free time courses for behavior, slow state of learning, and fast state of learning. Next, we computed the root mean squared errors (RMSEs) describing how well EM and LMSE recovered the hidden fast and slow states of learning and the overall behavior. Top: the RMSE for the behavioral fit (y), slow-state fit (x_s), and fast-state fit (x_f) are shown. Bottom: a relative RMSE metric was computed to compare the RMSEs of EM and LMSE fits to the same simulated behavior; the RMSE for the LMSE algorithm was divided by that of EM and multiplied by 100, and then a factor of 100 was subtracted to compute a percent increase of LMSE RMSE over that of EM. All bars represent the mean RMSE across 1,000 simulations. Error bars represent 95% confidence intervals. LMSE improved upon the RMSE of the behavior fit by ∼10% for all paradigms. However, EM was superior in uncovering the slow and fast states. The largest difference was observed for paradigm 3, followed by paradigm 1, then paradigms 2 and 4.

Fig. 5.

Parameter estimation errors. For each simulation, we computed the absolute value of the difference between each true parameter and the parameter values predicted by EM and LMSE. All bars represent the mean absolute parameter error across all simulations. Error bars represent 95% confidence intervals. For all parameters and paradigms, EM had lower estimation error than LMSE. P1–P4, paradigms 1–4, respectively.

Fig. 6.

Power analysis. We simulated within-subject and between-subject experiments to determine the number of subjects that would be required to detect a change in learning parameters. We created a pool of 1000 simulated subjects by sampling two-state model parameters from a multivariate normal distribution. We created different distributions by scaling a single learning parameter for each of the 1,000 subjects. We simulated behavior in paradigm 2 and fit the data with EM and LMSE. We then sampled subjects to perform hypothesis testing. For within-subject tests, we sampled the same subjects from different parameter levels. For between-subject tests, we sampled subjects independently from different parameter levels. For each test, we performed a paired t-test (within-subject analysis) or two-sample t-test (between-subject analysis) to determine whether EM or LMSE detected a statistically significant difference in the learning parameter. We repeated this process for different random samples of our subject population (10,000 for each test). Finally, we determined the minimum number of subjects that would be required for each algorithm to detect a significant difference for 85% of our samples. Here, we show the number of subjects required to reach an 85% detection rate for both EM (black) and LMSE (gray) as a function of the magnitude of the true parameter difference for each test (the effect size). We performed tests for both increases (solid lines with filled circles) and decreases (dashed lines with filled squares) in two-state model parameters. The results for the within-subject analysis and between-subject analysis are shown in A and B, respectively. We report only results for which <500 subjects were required to reach the 85% detection rate. For LMSE, >500 subjects were required for 5 different parameter-effect size pairs in the between-subject analysis. For EM, this occurred only once.

Fig. 7.

Sensitivity analysis for state and motor noise. A–C: we scaled the state and motor noise variances by 0.5, 2, 4, 6, 8, and 10 times the values reported in Table 1 (measured from our subject population). At each noise level, we performed 1,000 simulations of paradigm 2. We fit the simulated reaching behavior with EM and LMSE, generated EM and LMSE estimates of the behavior, fast, and slow states of learning, and finally computed the RMSE between the true time courses and model fits. The RMSEs for the behavior, slow, and fast state are shown in A, B, and C, respectively. Solid lines indicate the mean RMSE across all 1,000 simulations at each noise level. The shaded error bars indicate 95% confidence intervals around the mean. D–F: we performed another analysis where we allowed the slow and fast processes to have different variances. We fixed the overall level of state noise (sum of the fast- and slow-state variances) and performed a sensitivity analysis where we assigned different fractions of the overall state noise differentially to the slow and fast states. We tested levels where the slow (or fast) state had 25, 37.5, 50, 62.5, and 75% of the overall variance. For each level, we simulated 1,000 simulations of paradigm 2. We fit the simulated behavior using EM and LMSE and computed RMSEs for the behavior, slow process, and fast process, as in A, B, and C, respectively. The RMSEs for the behavior, slow, and fast state are shown in D, E, and F, respectively. Solid lines indicate the mean RMSE across all 1,000 simulations at each noise level. The shaded error bars indicate 95% confidence intervals around the mean.

Fig. 8.

Sensitivity analysis for the dynamics of the fast and slow states. We performed sensitivity analyses to determine how well EM and LMSE could isolate the fast and slow states of learning for two-state model parameters that differed from those observed for our visuomotor rotation subject population (Table 1). We analyzed one parameter at a time, fixing the remaining two-state model parameters to the values reported in Table 1. For each analysis, we scaled the two-state model parameter to several different values, corresponding to the effect sizes used in our power analysis in Fig. 6. At each parameter level, we performed 1,000 simulations of paradigm 2. We fit the simulated reaching behavior with EM and LMSE, generated EM and LMSE estimates of the behavior, fast, and slow states of learning, and finally computed the RMSE between the true time courses and model fits. The top, middle, and bottom rows, show the RMSE for the behavior, slow state, and fast state fits, respectively. The shaded error bars indicate 95% confidence intervals. The parameters investigated are as follows: the fast-state retention factor (A), the slow-state retention factor (B), the fast-state error sensitivity (C), and the slow-state error sensitivity (D). For each analysis, EM identified slow and fast states of learning with lower RMSE than LMSE. These results indicate that the relative difference between EM and LMSE performance would generalize to other dynamics of learning.

Fig. 9.

Comparison of EM and LMSE in a restricted parameter space. In our primary analysis, we found a preference for LMSE to assign slow retention factors that exceed 1, which led to unstable behavior of the predicted slow process. We asked whether LMSE could be rescued by modifying the parameter search space to prevent the identification of these unstable retention factors. To answer this question, we reanalyzed our simulations for paradigms 1–4 (see Figs. 3, 4, and 5) by refitting the EM and LMSE algorithm in a parameter space whose upper bounds for the slow and fast state retention factors were equal to 1. We used the EM and LMSE parameters to simulate noise-free time courses for behavior, slow state of learning, and fast state of learning. Next, we computed the RMSEs describing how well EM and LMSE recovered the hidden fast and slow states of learning and the overall behavior for the same simulations depicted in Fig. 4. Top, the RMSE for the behavioral fit (y), slow-state fit (x_s), and fast-state fit (x_f) are shown. Bottom: we computed the % difference between the RMSEs for EM and LMSE. Positive values indicate a larger RMSE for the LMSE algorithm. Error bars represent 95% confidence intervals. We found that restricting the upper bound on the slow- and fast-state retention factors improved the RMSE of the LMSE fits to the hidden states (compare Fig. 4 with Fig. 9) but did not completely rescue LMSE predictions; still, EM more accurately identified the true slow and fast states of learning.

Fig. 10.

Comparison of EM and LMSE on a trial-by-trial analysis of the data. We collected the behavior of n = 20 subjects in a visuomotor rotation task. We fit the two-state model to the trial-by-trial data recorded for individual subjects using EM and LMSE. We considered two trial-by-trial models that differed in terms of generalization. Top: the full-generalization model consisted of a single fast and slow state whose learning was completely generalized across targets. Bottom the no-generalization model consisted of separate fast and slow states for each of the 8 targets, whose learning did not generalize across targets. A: population behavior. We computed the average trial-to-trial behavior of the subject population. The average behavior (black) is shown overlaid with the average EM (blue) and average LMSE (red) fits. EM and LMSE had very similar fits to the behavior. B: predicted fast and slow states. For both the full and no-generalization models, EM estimated larger contributions from the slow state of learning and smaller contributions from the fast state of learning. Error bars indicate ± 1 SE. Here, the average time courses across the 8 fast states and 8 slow states are shown for the no-generalization model. C: we compared the corrected AIC of 2 competing likelihood models: one with state and motor noise and one without these noise sources. AICc was lower (better) for a model with state and motor noise. Here, we provide the mean difference in AICc for both models (state and motor noise likelihood – no state and motor noise likelihood). D: we used the trial-by-trial parameters to perform a set of control simulations. We simulated paradigm 2 a total of 1,000 times and fit each simulated data set with EM and LMSE. Whereas LMSE fit the observed reaching behavior more closely (y), EM vastly outperformed LMSE in the identification of the hidden slow and fast states (x_s and x_f).