Nonlinear convergence boosts information coding in circuits with parallel outputs

Abstract

Neural circuits are structured with layers of converging and diverging connectivity and selectivity-inducing nonlinearities at neurons and synapses. These components have the potential to hamper an accurate encoding of the circuit inputs. Past computational studies have optimized the nonlinearities of single neurons, or connection weights in networks, to maximize encoded information, but have not grappled with the simultaneous impact of convergent circuit structure and nonlinear response functions for efficient coding. Our approach is to compare model circuits with different combinations of convergence, divergence, and nonlinear neurons to discover how interactions between these components affect coding efficiency. We find that a convergent circuit with divergent parallel pathways can encode more information with nonlinear subunits than with linear subunits, despite the compressive loss induced by the convergence and the nonlinearities when considered separately.

Keywords: efficient coding; information theory; neural computation; retina; sensory processing.

Fig. 1.

Neural circuits are composed of inherently lossy components. (A) Schematic of retina circuit with its convergent and divergent structure. (B) Converging two inputs results in ambiguities. A two-input stimulus space is reduced to a single-output response space in which one response (B Upper, yellow and orange points) represents all stimuli along an isoline (B Lower, yellow and orange lines), where $s_1+s_2=$ constant. All entropy values shown are based on a discrete entropy computation (Materials and Methods). (C) Diverging a signal to two outputs can produce redundancies. (D) Nonlinear transformation of a Gaussian-distributed stimulus input with a ReLU can distort the distribution, producing a compressed response in which some portion of the stimulus information is discarded. (E and F) Convergent, divergent circuits with linear subunits (E) or nonlinear subunits (F). Subunit responses are weighted by $1/\sqrt{36}$. An example stimulus image is shown.

Fig. 2.

(A) The encoding of the stimulus space (top row) within each layer of a two-subunit convergent circuit configuration without divergence. Subunits (second row), summed subunits response distribution (third row), and nonlinear output response distribution (fourth row) are shown. (A Left) LSC. (A Right) NSC. The output nonlinearity does not have an additional effect on the summed nonlinear subunits without noise. (B) Histograms of the output response for the NSC are shown for configurations with 3, 8, and 15 subunits. The subunit responses are normalized so that each subunit is weighted by $1/\sqrt{N}$, where $N$ is the number of subunits. The inputs $\left[s1,s2,\dots ,sN\right]$ are independently drawn from a Gaussian distribution (Materials and Methods). (C) Normalized entropy of output response as a function of number of convergent subunits where subunits are normalized as in B and the circuit entropy is normalized by the entropy of the summed linear subunits (Materials and Methods). Black curve, NSC; gray, LSC; dark pink, nonlinear subunits with optimized sigmoidal output nonlinearity; light pink, linear subunits with optimized sigmoidal output nonlinearity. SD of entropy over 10 runs for each configuration is on the order of between $10^{-4}$ and $10^{-2}$ bits.

Fig. 3.

Visualization of stimulus and response mappings at each level of a convergent, divergent circuit with two inputs, two subunits for each pathway (ON and OFF pathways), and a nonlinear output neuron for each pathway. The points in all subsequent plots are color-coded by the stimulus quadrant from which they originate. (A) The stimulus space (Upper) has color-coded quadrants. The two-input stimulus space maps onto a 2D linear subunit space for each pathway (A Lower Left, ON; A Lower Right, OFF). The subunit spaces are shown before subunit normalization. (B and C) The response space is shown for the linear sum of subunits before the output nonlinearity is applied (B) and after the nonlinear output response (C). (D) The two-input stimulus space (Upper) maps onto a 2D nonlinear subunit space for each pathway (Lower Left, ON; Lower Right, OFF). (E) The output response space for the NSC. Note that the output response before the output nonlinearity is applied (not shown) is identical to the output response after the output nonlinearity is applied for the circuit with nonlinear subunits. (F) Normalized entropy of the output response for convergent, divergent circuits with increasing input and subunit dimension (subunit responses are normalized as before). The circuit entropy is normalized by the entropy of the summed linear subunits. Gray, LSC in C; black, NSC in E; light pink, LSC with optimal sigmoidal output nonlinearity; dark pink, NSC with optimal sigmoidal output nonlinearity.

Fig. 4.

Mean and contrast encoding of convergent, divergent circuits from Fig. 3 (two inputs, ON and OFF outputs). (A) Visualization of the stimulus mean and output response spaces. For example, the bright mean stimulus band contains the two-input image samples that have the highest mean luminance. The red square is an arbitrary reference point. In the stimulus space, the cyan square has the same mean luminance as the red square, but a different contrast, while the red circle has the same contrast as the red square, but a different mean luminance. (B) Visualization of the stimulus contrast and output response spaces. The high-contrast stimulus bands contain two-input image samples that have high contrast, whereas the low-contrast band contains two-input image samples where the input luminance is more correlated.