Variation in sequence dynamics improves maintenance of stereotyped behavior in an example from bird song

Abstract

Performing a stereotyped behavior successfully over time requires both maintaining performance quality and adapting efficiently to environmental or physical changes affecting performance. The bird song system is a paradigmatic example of learning a stereotyped behavior and therefore is a good place to study the interaction of these two goals. Through a model of bird song learning, we show how instability in neural representation of stable behavior confers advantages for adaptation and maintenance with minimal cost to performance quality. A precise, temporally sparse sequence from the premotor nucleus HVC is crucial to the performance of song in songbirds. We find that learning in the presence of sequence variations facilitates rapid relearning after shifts in the target song or muscle structure and results in decreased error with neuron loss. This robustness is due to the prevention of the buildup of correlations in the learned connectivity. In the absence of sequence variations, these correlations grow, due to the relatively low dimensionality of the exploratory variation in comparison with the number of plastic synapses. Our results suggest one would expect to see variability in neural systems executing stereotyped behaviors, and this variability is an advantageous feature rather than a challenge to overcome.

Keywords: maintenance learning; motor learning; reinforcement learning; skilled movement; songbird.

Fig. 1.

Model of bird song learning. (A) Schematic of the avian song system. (B) Schematic of learning model. Learning occurs on the synaptic weights from HVC (n = 500) to RA (n = 48). (C) Schematic of HVC firing patterns and perturbation events. (D) Learning trajectory. Error is defined as the absolute difference between the target m₁ and m₂ and the model output m₁ and m₂. Red trace shows unperturbed trajectory. Black trace shows trajectory with 50 HVC perturbation events. Blue dot on y axis indicates the error before learning. (Inset) Three perturbation events. (E) The difference between initial and final error in learning trajectory as a function the number of HVC perturbation events per 10⁵ iterations averaged over 50 trials. Error reduction is defined as difference between the first iteration and average of the last 500 iterations for each trial. HVC perturbations slightly decrease error reduction. Error bars represent standard error.

Fig. 2.

Tests of robustness. (A) Schematic RA cell loss. After a subset of RA is removed, the song is performed, and the new error is computed. (B) Error in song as a function of number of HVC perturbations per 10⁵ iterations for the full RA network and for subpopulations of RA. Arrows indicate error from full RA network to error from RA network after cell loss. Error is averaged over randomly drawn subsets of RA and over trials. HVC perturbations increase the ability of a subpopulation of the RA network to represent song. (C) Target song shift. After 10⁵ iterations, the target song is altered, and relearning proceeds for 3,000 iterations. (D) Relearning trajectory after target song changes. (E) Number of iterations to half-decay of learning trajectory. HVC perturbations speed up the adaptation process with minor penalty in final error. Error bars represent standard error.

Fig. 3.

Mechanisms underlying robustness. (A) Example initial and final W matrices after 10⁵ iterations of learning with HVC index ordered by when the HVC cell bursts. (Top) Initial W for 0 and 50 HVC perturbations; (Middle) final W for 0 (Left) and 50 (Right) HVC perturbation events; (Bottom) learned motor trajectories for 0 (Left) and 50 (Right) HVC perturbations. Black line is the target. Light colored lines are learned trials after 10⁵ iterations. (B) Average pairwise correlations after 10⁵ iterations of learning between HVC cells’ synaptic projections to RA (columns of W) as a function of the time difference in firing onset. Averages are taken over HVC cells and learning trials. Correlations decrease with more HVC perturbation events. (C) Maximum pairwise correlations between HVC neurons’ synaptic projections to RA as a function of the number of HVC perturbation events. (Inset) Trajectory of maximum correlations over learning iterations. Drops in correlation occur at HVC perturbation events. (D) Schematic of the progression of W over the course of learning. Error first proceeds quickly to a minimum. Correlations in the LMAN inputs then slowly push the solution toward higher correlations if learning continues without perturbations. Solid white line shows region of approximately equal error. Purple trajectory shows W matrix undergoing learning from an initial position of low pairwise correlation. Gold trajectory shows W matrix undergoing learning from an initial position of high pairwise correlation. Blue and green dots show initial and final positions. The region near and at W_ij = W_i,j+1 where error increases again is not shown in this schematic. (E) Average relearning trajectory (over 50 trials) for altered song target. (Inset) Maximum HVC pairwise correlation in W at the beginning and end of 3,000 song iterations for representative trials from E. The correlations of the initial weight matrix before learning strongly influences the correlations of the final weights after learning. Error bars represent standard error.

Fig. 4.

Model comparisons of perturbing HVC activity. (A) HVC perturbation scheme wherein a subset of HVC cells are silenced and reactivated at each perturbation event. Synapses weaken in silent cells. (B) The same scheme as in A, but without synaptic weakening. (C) HVC perturbation scheme wherein a subset of HVC cells’ firing times randomly shift at each perturbation event. (D–G) Comparison of performance and robustness metrics across perturbation schemes and frequency. (D) Error improvement over 10⁵ iterations. Minor cost is incurred for increased perturbation frequency. (Same as in Fig. 1E.) (E) Number of iterations to half decay of relearning trajectory. (Same as in Fig. 2E.) (F) Increase in error due to RA cell loss. Gray indicates performance error with full RA network. Other colors indicate performance error with partial RA network. (Same as in Fig. 2B.) (G) Maximum pairwise correlations between HVC neurons’ synaptic projections to RA. Error bars represent standard error.