Stochastic activity in low-rank recurrent neural networks

Abstract

The geometrical and statistical properties of brain activity depend on the way neurons connect to form recurrent circuits. However, the link between connectivity structure and emergent activity remains incompletely understood. We investigate this relationship in recurrent neural networks with additive stochastic inputs. We assume that the synaptic connectivity can be expressed in a low-rank form, parameterized by a handful of connectivity vectors, and examine how the geometry of emergent activity relates to these vectors. Our findings reveal that this relationship critically depends on the dimensionality of the external stochastic inputs. When inputs are low-dimensional, activity remains low-dimensional, and recurrent dynamics influence it within a subspace spanned by a subset of the connectivity vectors, with dimensionality equal to the rank of the connectivity matrix. In contrast, when inputs are high-dimensional, activity also becomes potentially high-dimensional. The contribution of recurrent dynamics is apparent within a subspace spanned by the totality of the connectivity vectors, with dimensionality equal to twice the rank of the connectivity matrix. Applying our formalism to excitatory-inhibitory networks, we discuss how the input configuration also plays a crucial role in determining the amount of amplification generated by non-normal dynamics. Our work provides a foundation for studying activity in structured brain circuits under realistic noise conditions, and offers a framework for interpreting stochastic models inferred from experimental data.

Figure 1:

Setup. A. Left: model architecture. Right: sample activity traces from three randomly chosen neurons. B. Rank-one recurrent connectivity. C. Illustration of a sample activity trajectory in the high-dimensional space where each axis corresponds to the activity of a different neuron. Activity (black arrow) is given by the sum of two components (Eq. 7, blue arrows); the direction of the component generated from recurrent interactions is fixed, and is aligned with the connectivity vector $\mathit{m}$.

Figure 2:

Rank-one RNN receiving one-dimensional stochastic inputs. A. Model architecture. B. Activity covariance is low-dimensional, and is spanned by connectivity vector $\mathit{m}$ together with the external input vector $\mathit{u}$. As a consequence, activity is contained within the plane collinear with these two vectors. C–D–E. Example of a simulated network with ${\rho }_{nu}=0$. In C: covariance spectrum. Components larger than 10 are not displayed (they are all close to zero). In D: overlap between the dominant principal components estimated from simulated activity and the theoretically-estimated PCs (left), or the vectors $\mathit{m}$ and $\mathit{u}$ (right). Overlaps are quantified via Eq. 4, with input vectors $\mathit{u}$ chosen to be normalized. Note that here, but not in G, only one principal component can be identified. In E: simulated activity projected on the two dominant PCs. F–G–H. Same as in C–D–E, example with ${\rho }_{nu}>0$.

Figure 3:

Rank-one RNN receiving high-dimensional stochastic inputs. A. Model architecture. B. The activity covariance is high-dimensional, with all eigenvalues taking identical values except for two – one larger and one smaller. The principal components associated with these two eigenvalues lie within the plane spanned by the connectivity vectors $\mathit{m}$ and $\mathit{n}$. C. Covariance eigenvalues as a function of overlap between connectivity vectors. The dashed vertical line indicates the value of ${\rho }_{mn}$ for which dynamics becomes unstable. Black arrows indicate the value of ${\rho }_{mn}$ that is used for simulations in Fig. 4. D. Dimensionality. Horizontal black lines indicate the maximum ($N$) and the minimum (1) possible values. E. Components of ${\mathit{v}}_{+}$ (or PC1 vector, left) and ${\mathit{v}}_{-}$ (or PCN vector, right) along connectivity vectors $\mathit{m}$ and $\mathit{n}$, as from Eq. 24. F. Overlap (Eq. 4) between the principal components ${\mathit{v}}_{+}$ and ${\mathit{v}}_{-}$ (after normalization) and the connectivity vectors $\mathit{m}$ and $\mathit{n}$.

Figure 4:

Rank-one RNN receiving high-dimensional stochastic inputs. A–B–C. Example of a simulated network with ${\rho }_{mn}=-0.5$. In A: covariance spectrum. In B: overlap between two principal components (the strongest and the weakest) estimated from simulated activity and the theoretically-estimated vectors ${\mathit{v}}_{+}$ and ${\mathit{v}}_{-}$ (top), or vectors $\mathit{m}$ and $\mathit{n}$ (bottom). Overlaps are quantified via Eq. 4. In C: simulated activity projected on two different pairs of PCs. D–E–F. Same as in A–B–C, example with ${\rho }_{mn}=0.3$. Note that, although the qualitative behaviour of activity in the two examples is similar, activity in the example network in A–B–C is overall higher dimensional.

Figure 5:

Stochastic activity in rank-two recurrent neural networks. A. Rank-two connectivity. B. Eigenvalues of the covariance matrix that are different than the reference value ${\mu }_{\mathrm{r}\mathrm{e}\mathrm{f}}$. As connectivity is rank-two, four eigenvalues are perturbed; we sort them in ascending order. Violin plots show the distribution of perturbed eigenvalues for different values of the parameters ${\rho }_{m^1{n}^1}$ and ${\rho }_{m^2{n}^2}$. Note that, for all sets of parameters, two eigenvalues are increased and two decreased with respect to ${\mu }_{\mathrm{r}\mathrm{e}\mathrm{f}}$. C. Dimensionality as a function of ${\rho }_{m^1{n}^1}$ and ${\rho }_{m^2{n}^2}$. The black dashed lines indicate the parameter values for which dynamics becomes unstable. The tiny black square indicates the parameter values that are used for simulations in F–G. In both B and C, we keep the values of ${\rho }_{m^1{m}^2}$ and ${\rho }_{n^1{n}^2}$ fixed to zero (see Methods 7). D–E. Same as for B–C, but for a different parametrization, where we keep ${\rho }_{m^1{n}^1}$ and ${\rho }_{m^2{n}^2}$ fixed to zero. F–G. Example of a simulated network, parameters indicated in C. In F: covariance spectrum. In G: overlap between four selected principal components (the strongest and the weakest) estimated from simulated activity and the theoretically-estimated covariance eigenvectors (left) and the connectivity vectors (right). Overlaps are quantified via Eq. 4. The theoretical expressions for this case are reported in Methods 7.

Figure 6:

Stochastic activity in a rank-one excitatory-inhibitory circuit. A. E-I circuit with high-dimensional inputs. B. Variance explained by PC1 (top) and PC2 (bottom) as a function of the overall recurrent connectivity strength $w$ and the relative dominance of inhibition $g$. In B–C–D, the black solid line separates the regions for which the non-zero eigenvalue $\lambda$ is larger or smaller than one. The black dotted line separates the regions for which the non-zero eigenvalue $\lambda$ is larger or smaller than zero. Note the different color scales in the top and bottom plots. C. Overlap between PC1 and the sum (top) and diff (bottom) directions. D. Non-zero eigenvalue of the synaptic connectivity matrix $\lambda =w(1-g)$. E. E-I circuit with one-dimensional inputs. F. Variance explained by PC1 (top) and PC2 (bottom) as a function of the overall recurrent connectivity strength $w$ and the direction of the input vector $\mathit{u}$. The input direction is parametrized by an angle $\theta$ (see Methods 8), so that $\theta =0^{\circ }$ (resp. 90°) correspond to inputs entering only E (resp. I), while $\theta ={45}^{\circ }$ (resp. 135°) corresponds to inputs aligned with the sum (resp. diff) direction. G. Variance explained by PC1 (top) and PC2 (bottom) as a function of the relative dominance of inhibition $g$ and the direction of the input vector $\mathit{u}$. H. Overlap between PC1 and the sum (top) and diff (bottom) direction.