Geometry of population activity in spiking networks with low-rank structure

Abstract

Recurrent network models are instrumental in investigating how behaviorally-relevant computations emerge from collective neural dynamics. A recently developed class of models based on low-rank connectivity provides an analytically tractable framework for understanding of how connectivity structure determines the geometry of low-dimensional dynamics and the ensuing computations. Such models however lack some fundamental biological constraints, and in particular represent individual neurons in terms of abstract units that communicate through continuous firing rates rather than discrete action potentials. Here we examine how far the theoretical insights obtained from low-rank rate networks transfer to more biologically plausible networks of spiking neurons. Adding a low-rank structure on top of random excitatory-inhibitory connectivity, we systematically compare the geometry of activity in networks of integrate-and-fire neurons to rate networks with statistically equivalent low-rank connectivity. We show that the mean-field predictions of rate networks allow us to identify low-dimensional dynamics at constant population-average activity in spiking networks, as well as novel non-linear regimes of activity such as out-of-phase oscillations and slow manifolds. We finally exploit these results to directly build spiking networks that perform nonlinear computations.

Fig 1. Low-rank connectivity and state space dynamics.

A: Illustration of recurrent neural network architecture, consisting of inputs and recurrent connectivity. B: Representation of inputs and connectivity in terms of vectors. The input weights form an input vector I. In spiking networks, the recurrent connectivity is composed of a sparse excitatory-inhibitory part (zero entries in white, excitatory connections in red, inhibitory in blue) and a low-rank structure defined by pairs of connectivity vectors m and n. The illustration shows a unit-rank example (R = 1). C: Left: Spike times of three neurons in the spiking network. Right: dynamics of instantaneous firing rates computed from spikes using an exponential filter with timescale τ_f = 100ms. D: Three-dimensional illustration of low-dimensional dynamics in the activity state space where each axis represents the firing rate of one neuron. In a unit-rank network, the activity is expected to be confined to a two-dimensional plane spanned by the vectors, m and I. We refer to the direction (1, 1,…1) as the global axis (orange). E: Projections of activity on two axes: (top) global axis corresponding to the population-averaged firing rate; (bottom) axis defined by the input vector I.

Fig 2. Low-dimensional dynamics generated by external inputs in rate networks with low-rank connectivity.

A: Illustration of the geometry in the activity state-space. The input vector I, the connectivity vectors m and n (green), and the global axis (1, 1, …, 1) (orange) define a set of directions and a subspace within which the low-dimensional dynamics unfold. The overlaps of the vectors I and m with the global axis predict whether inputs give rise to a change in the population-averaged activity. The overlap of I with n instead determines whether an input engages recurrent activity along the direction m. The three columns display three different arrangements of the input vector (depicted in a different color in each column). Left: I aligned with the global axis; middle: I orthogonal to both n and the global axis; right: I aligned with n, but I and m orthogonal to the global axis. B: Input vector is multiplied by a scalar u(t) which is a step function from t = 1s. C: Individual firing rates r_i(t) for a subset of 10 neurons in each network. D: Population firing rate, averaged over all neurons in the network. E: Projections of the firing rate trajectory r(t) onto the (I,m) plane. F: PCA analysis of the firing rate dynamics r(t). Variance explained by each of the first 8 PCs. Inserts: Projections of the first two PCs onto the global axis, the input vector I and the connectivity vector m. The connectivity vectors m and n have a zero mean and unit standard deviation, and are orthogonal to each other. Vectors n and I are orthogonal except in blue where the overlap is σ_nI = 1. Vectors I in gray and blue have a zero mean and unit standard deviation, while vector I in purple is along the global axis. Network parameters are given in Table 1.

Fig 3. Low-dimensional dynamics generated by external inputs in spiking networks with low-rank structure.

A: Illustration of the geometry of input (varying color) and connectivity vectors (green) with respect to the global axis (orange). Left: input vector I along the global axis; middle: input vector I orthogonal to n; right: input vector I along the vector n. B: Input vector is multiplied by a scalar u(t) which is a step function from t = 1s. C: Raster plot showing action potentials for a subset of 30 neurons out of N = 12500 in each network. D: Population firing rate obtained by averaging instantaneous firing rates of all neurons. E: Projections of the firing rate trajectory r(t) onto the (I, m) plane. F: PCA analysis of firing rate dynamics r(t). Variance explained by each of the first 8 PCs. Inserts: Projections of the first 3 PCs onto the global axis (first row), and vectors I and m. The connectivity vectors m and n have a zero mean and unit standard deviation, and are orthogonal to each other. Vectors n and I are orthogonal except in blue where the overlap is n^TI/N = 0.4mV². Vectors I in gray and blue have a zero mean and unit standard deviation, while vector I in purple is along the global axis. All analyses were performed on instantaneous firing rates computed using a filter timescale of τ_f = 100ms. Network parameters are given in Table 2.

Fig 4. Low-dimensional dynamics generated by external inputs in spiking network in which the full connectivity is sparse and satisfies Dale’s law.

The connectivity consisted of the superposition of the random term J^EI and a sparsified unit-rank part P, in which we set to zero entries for which $J_{ij}^{EI}=0$. The input vector I is along the vector n. A: Raster plot showing action potential for a subset of 30 neurons. B: Population firing rate obtained by averaging instantaneous firing rates of all neurons. C: Projection of the firing rate trajectory r(t) onto the (I, m) plane. D: PCA analysis of firing rate dynamics r(t). Variance explained by each of the first 8 PCs parameters in Table 2.

Fig 5. Influence of firing regime and filter timescales on low-dimensional dynamics in spiking networks.

A-C: Synchronous irregular (SI) regime. A: Top: Raster plot showing action potentials for a subset of 30 neurons in the network. Bottom: population-averaged firing rate computed using filter time constants of 1ms (blue) and 100ms (orange). B: PCA analysis of trajectories of instantaneous firing rates computed from spike trains using two different filter time constants (top: 1ms, bottom: 100ms). Main panels: variance explained by each of the first 8 PCs; inserts: projections of the first 3 principal components on the global vector, I and m. C: Top: Projections of the firing rate trajectories on the plane defined by vectors m and I. Bottom: Projection of the first principal component on the global axis (black) and on the vector m (green) as a function of the filter time constant. D-F: Similar to A-C, for the network in asynchronous irregular regime shown in the right column of Fig 3. G-I Similar to A-C, for a network without the background E-I connectivity. The firing regime was controlled by varying the inhibition strength in the random EI connectivity, the baseline input and synaptic delays (see Table 2). The unit-rank connectivity structure was identical to Fig 3 right column, with zero-mean input and connectivity vectors. At time t = 1s, a step input was given along the input vector I that was aligned with n. Network parameters are given in Table 2.

Fig 6. Nonlinear autonomous activity in networks with unit-rank connectivity structure.

A-F: Rate networks. A-C: Connectivity vectors m and n with non-zero means 〈m〉, 〈n〉, and zero covariance σ_mn. A: Fixed points of the collective variable κ as a function of the overlap n^Tm/N, low (black) and high (red) activity state. Insert: RHS of the equation dκ/dt (Eq (6)), κ (yellow) and 〈n〉〈ϕ〉(κ) (gray), shown for the overlap n^Tm/N = 10. Fixed points (red dots) correspond to the intersections of κ and 〈n〉ϕ(κ) which is a positive function. The bifurcation therefore leads to a low and a high state. B: Illustration of the single-unit firing rates in the two states when n^Tm/N = 10 (dashed line in A, green) for 100 units. Top: low activity state. Bottom: high activity state. C: Population-averaged firing rate as a function of n^Tm/N. D-F: same as A-C, for connectivity vectors m and n with zero means 〈m〉, 〈n〉, and non-zero covariance σ_mn. D: Fixed points of the collective variable κ as a function of the overlap n^Tm/N. Insert: RHS of the equation dκ/dt (Eq (6)), κ (yellow) and κ〈ϕ′〉(κ) (gray), shown for the overlap n^Tm/N = 11.2. Fixed points (red dots) correspond to the intersection of κ and κ〈ϕ′〉(κ), which is symmetric around the y axis. The bifurcation therefore leads to two symmetric states (red and blue) on top of the low activity state. E: Illustration of the single-unit firing rates in the two symmetric states. F: Population-averaged firing rate as a function of n^Tm/N. G-L: Simulations of the spiking network. G-I: connectivity vectors m and n with non-zero means 〈m〉, 〈n〉 and zero covariance σ_mn. G: bifurcation to low and high states as 〈n〉 is increased. H: raster plots of the spiking activity in the two states when n^Tm/N = 1.35mV (dashed line in J, green) for 20 neurons. Top: activity of 20 neurons in the high state. Bottom: activity of all (12500) neurons in the low state. The activity in the low state is highly sparse [7]. I: population-averaged firing rate in the two states. J-L: same as G-I connectivity vectors m and n with zero means 〈m〉, 〈n〉 and non-zero covariance σ_mn. J: bifurcation to two symmetric states as σ_mn is increased. K: raster plots of the spiking activity in the two states when n^Tm/N = 32mV (dashed line in J, green) for 20 neurons. L: population-averaged firing rate in the two states. Dots: simulations, lines: Monte Carlo integration predictions. Network parameters are shown in Tables 3 and 4.

Fig 7. Nonlinear dynamics in networks with rank-two structure.

A-F: Connectivity structure with two complex-conjugate eigenvalues. A-C: Rate networks. A: Top: Illustration of the single-unit firing rates for the first 100 neurons. Bottom: Population-averaged firing rate. B: Projections of the firing rates r(t) on the m⁽¹⁾ − m⁽²⁾ plane. Insert: overlap matrix. C: Projections of the firing rates r(t) on vectors m⁽¹⁾ and m⁽²⁾ as a function of time. D-F: Analogous to (A-C), spiking network. D: Top: raster plots of the spiking activity for first 50 neurons. Bottom panel: population firing rate. E: Projections of the firing rates r(t) on the m⁽¹⁾ − m⁽²⁾ plane. Insert: overlap matrix. F: Projections of the firing rates r(t) on vectors m⁽¹⁾ and m⁽²⁾ as a function of time. (G-I) Rate network dynamics for an overlap matrix that has two real, degenerate eigenvalues. G: Top panel: illustration of the single-unit firing rates for the first 100 neurons. Bottom panel: Population firing rate. H: Projections of the firing rates r(t) on the m⁽¹⁾ − m⁽²⁾ plane. Insert: overlap matrix. I: Projections of the firing rate r(t) on vectors m⁽¹⁾ and m⁽²⁾ as a function of time. (J-L) same analysis as in (G-I) for a spiking model. J: Top panel: raster plots of the spiking activity for first 50 neurons. Bottom panel: population firing rate. K: Projections of the firing rates r(t) on the m⁽¹⁾ − m⁽²⁾ plane. Insert: overlap matrix. L: Projections of the firing rate r(t) on vectors m⁽¹⁾ and m⁽²⁾ as a function of time. Different colors in the middle column (B,E,H,K) corresponds to network instances with different connectivity vectors but identical statistics. Network parameters are shown in Tables 5 and 6.

Fig 8. Nonlinear dynamics in networks outside of the AI regime.

A,C,E: Network operating in the SI regime. A: nonlinear dynamics in rank-one network. Left, top: fixed points of the collective variable κ as a function of the overlap n^Tm/N, for connectivity vectors m and n with zero means 〈m〉, 〈n〉, and non-zero covariance σ_mn. Left, bottom: population averaged firing rate as a function of n^Tm/N. Right: raster plots in the two symmetric states. C: non-linear dynamics in rank-two network, connectivity structure with two complex-conjugate eigenvalues. Left, top: raster plot of the spiking activity for first 50 neurons. Left, bottom: population firing rate. Right: projections of the firing rates r(t) on the m⁽¹⁾ − m⁽²⁾ plane. E: non-linear dynamics in rank-two network for an overlap matrix that has two real, degenerate eigenvalues. Left, top: raster plots of the spiking activity for first 50 neurons. Left, bottom: population firing rate. Right: projections of the firing rates r(t) on the m⁽¹⁾ − m⁽²⁾ plane. B,D,F: same as A,C,E for network without the background E-I connectivity.

Fig 9. Spiking network implementation of the perceptual decision-making task.

A: Top panel: two instances of the fluctuating input signal with a positive (orange) and a negative (blue) mean. Bottom panel: network readout of the activity generated by the two inputs. B: Raster plots for the first 50 neurons. C: Population firing rate. D: Dynamics projected onto the I − m plane. E: Psychometric function showing the fraction of positive outputs at different values of the overlap σ_nm. Orange color corresponds to positive ($\overline{u}=0.512$), while blue to negative mean-input ($\overline{u}=-0.512$). Parameters: N = 12500, σ_u = 1, σ_nI = 0.26, σ_mw = 2.1, σ_m² = 0.02, τ_f = 100ms.