Local Homeostatic Regulation of the Spectral Radius of Echo-State Networks

Abstract

Recurrent cortical networks provide reservoirs of states that are thought to play a crucial role for sequential information processing in the brain. However, classical reservoir computing requires manual adjustments of global network parameters, particularly of the spectral radius of the recurrent synaptic weight matrix. It is hence not clear if the spectral radius is accessible to biological neural networks. Using random matrix theory, we show that the spectral radius is related to local properties of the neuronal dynamics whenever the overall dynamical state is only weakly correlated. This result allows us to introduce two local homeostatic synaptic scaling mechanisms, termed flow control and variance control, that implicitly drive the spectral radius toward the desired value. For both mechanisms the spectral radius is autonomously adapted while the network receives and processes inputs under working conditions. We demonstrate the effectiveness of the two adaptation mechanisms under different external input protocols. Moreover, we evaluated the network performance after adaptation by training the network to perform a time-delayed XOR operation on binary sequences. As our main result, we found that flow control reliably regulates the spectral radius for different types of input statistics. Precise tuning is however negatively affected when interneural correlations are substantial. Furthermore, we found a consistent task performance over a wide range of input strengths/variances. Variance control did however not yield the desired spectral radii with the same precision, being less consistent across different input strengths. Given the effectiveness and remarkably simple mathematical form of flow control, we conclude that self-consistent local control of the spectral radius via an implicit adaptation scheme is an interesting and biological plausible alternative to conventional methods using set point homeostatic feedback controls of neural firing.

Keywords: echo-state networks; homeostasis; recurrent networks; reservoir computing; spectral radius; synaptic scaling.

Figure 1

Online spectral radius regulation using flow control. The spectral radius R_a and the respective local estimates R_{a, i} as defined by (10). For the input protocol see section 5.3. (A) Dynamics of $R_{\text{a},i}^2$ and $R_{\text{a}}^2$, in the presence of heterogeneous independent Gaussian inputs. Local adaptation. (B) Distribution of eigenvalues of the corresponding effective synaptic matrix ${\hat{W}}_{\text{a}}$, after adaptation. The circle denotes the spectral radius.

Figure 2

Online spectral radius regulation using variance control. The spectral radius R_a and the respective local estimates R_{a, i} as defined by (10). For the input protocol see section 5.3. (A) Dynamics of $R_{\text{a},i}^2$ and $R_{\text{a}}^2$, in the presence of heterogeneous independent Gaussian inputs. Local adaptation. (B) Distribution of eigenvalues of the corresponding effective synaptic matrix ${\hat{W}}_{\text{a}}$. The circles denote the respective spectral radius.

Figure 3

XOR performance for flow control. Color-coded performance sweeps for the XOR-performance (18) after adaptation using flow control. Averaged over five trials. The input has variance ${\sigma }_{\text{ext}}^2$ and the target for the spectral radius is R_t. (A,B) Heterogeneous binary/Gaussian input protocols. Optimal performance for a given σ_ext was estimated as a trial average (yellow solid line) and found to be generally close to criticality, R_a = 1, as measured (white dashed lines).

Figure 4

XOR performance for variance control. Color-coded performance sweeps for the XOR-performance (18) after adaptation using variance control. Averaged over five trials. The input has variance ${\sigma }_{\text{ext}}^2$ and the target for the spectral radius R_t. (A,B) Heterogeneous binary/Gaussian input protocols. Optimal performance (yellow solid line) is in general close to criticality, R_a = 1, as measured (white dashed lines).

Figure 5

Size dependence of correlation. Comparison between the variance ${\sigma }_{\text{bare}}^2$ of the bare recurrent input $x_{\text{bare}}=\sum _j{W}_{ij}{y}_j$ with ${\sigma }_{\text{w}}^2{\sigma }_{\text{y}}^2$. Equality is given when the presynaptic activities are statistically independent. This can be observed in the limit of large network sizes N for uncorrelated input data streams (homogeneous and heterogeneous Gaussian input protocols), but not for correlated inputs (homogeneous and heterogeneous binary input protocols). Compare section 5.3 for the input protocols. Parameters are σ_ext = 0.5, R_a = 1, and μ_t = 0.05.

Figure 6

Input induced activity correlations. For heterogeneous binary and Gaussian inputs (A,B), the dependency of mean activity cross correlations $\overline{C}$ (see Equation 21). $\overline{C}$ is shown as a function of the spectral radius R_a. Results are obtained for N = 500 sites by averaging over five trials, with shadows indicating the standard error across trials. Correlations are due to finite-size effect for the autonomous case σ_ext = 0.

Figure 7

Convergence time with and without adaptation rate renormalization Number of time steps T_conv needed for $|R_{\text{a}}(t)-R_{\text{a}}(t-1){|}^2$ to fall below 10⁻³. Shown are results using heterogeneous Gaussian input without and with, (A) and respectively (B), a renormalization of the learning rate ${ϵ}_a\to {ϵ}_{\text{a}}/{\overline{x}}_{\text{r}}^2$. Note that, due to computational complexity, an estimate of R_a given by (10) was used. An initial offset of 0.5 from the target R_t was used for all runs. Color coding of R_t is the same in both panels.

Figure 8

Spectral radius adaptation dynamics. The dynamics of the synaptic rescaling factor a and the squared activity ${\sigma }_{\text{y}}^2$ (orange), as given by (6). Also shown is the analytic approximation to the flow (blue), see (33) and (34), and the respective nullclines Δa = 0 (green) and $\Delta {\sigma }_{\text{y}}^2=0$ (red). For the input, the heterogeneous binary protocol is used. (A–D) Correspond to different combinations of external input strengths and target spectral radii. The black dots show the stead-state configurations of the simulated systems. ϵ_a = 0.1.

Figure 9

Variance control for the spectral radius. The spectral radius R_a, given by the approximation $R_{\text{a}}^2=\sum _i{a}_i^2/N$, for the four input protocols defined in section 5.3. Lines show the numerical self-consistency solution of (49), symbols the full network simulations. Note the instability for small σ_y and σ_ext. (A) Homogeneous independent Gaussian input. (B) Homogeneous identical binary input. (C) Heterogeneous independent Gaussian input. (D) Heterogeneous identical binary input.

Figure 10

Phase transition of activity variance Shown are solutions of the analytical approximation given in (55), capturing the onset of activity (characterized by its variance ${\sigma }_{\text{y}}^2$) at the critical point R_a = 1.