Nonlinear decoding of a complex movie from the mammalian retina

Abstract

Retina is a paradigmatic system for studying sensory encoding: the transformation of light into spiking activity of ganglion cells. The inverse problem, where stimulus is reconstructed from spikes, has received less attention, especially for complex stimuli that should be reconstructed "pixel-by-pixel". We recorded around a hundred neurons from a dense patch in a rat retina and decoded movies of multiple small randomly-moving discs. We constructed nonlinear (kernelized and neural network) decoders that improved significantly over linear results. An important contribution to this was the ability of nonlinear decoders to reliably separate between neural responses driven by locally fluctuating light signals, and responses at locally constant light driven by spontaneous-like activity. This improvement crucially depended on the precise, non-Poisson temporal structure of individual spike trains, which originated in the spike-history dependence of neural responses. We propose a general principle by which downstream circuitry could discriminate between spontaneous and stimulus-driven activity based solely on higher-order statistical structure in the incoming spike trains.

Fig 1. Linear decoding of a complex movie.

A: An example stimulus frame. At each site (red dots = partially shown 20×20 grid) the stimulus was convolved with a spatial gaussian filter (red circle = 1σ). Typical RGC receptive field center size shown in gray. B: Responses of 91 RGCs with 750 ms decoding window overlaid in blue. C: Three example luminance traces (red) and the linear decoders’ predictions (blue). D: Decoded frame (same as in A) reconstructed from 20×20 separately decoded traces. Disc contours of the original frame shown for reference in green. E: RF centers of the 91 cells (black dots = centers of fitted ellipses). RF centers overlapping a chosen site (red dot) are highlighted in blue. F: Performance of the linear decoders across space, as Fraction of Variance Explained (FVE). Black dots as in E; black contour is the boundary FVE = 0.4. G: Performance of the linear decoders (FVE) across sites as a function of cell coverage (grayscale = conditional histograms, red dots = means, error bars = ± SD). H: Average decoding error across sites (MSE ± SD) of 10-disc-trained decoders, tested on withheld stimuli with different numbers of discs. I: Cells (black dots = RF center positions) contributing to the decoding at two example sites (red circles); decoding filters shown below. For each site, contributing cells (highlighted in red and joined to the site) account for at least half of the total L1 norm. J: Decoding field of a single cell (here, evaluated over a denser 50×50 grid and normalized to unit maximal variance); the cell’s RF center shown in black.

Fig 2. Nonlinear decoding outperforms linear decoding.

A: Luminance trace (red) with linear (blue) and nonlinear KRR (green) and neural network (grey) predictions. B: Average decoder performance (± SD across sites), achievable using increasing numbers of cells with highest L1 filter norm. For nonlinear decoding, “All” is the optimal subset that maximizes performance (S7 Fig). Since the neural network (grey point with an error bar) simultaneously decodes the movie at all sites, it only makes sense to train it using “All” cells. C: Average ROC across all testing movie frames. D: Fractional improvement (average ± SEM across sites) of nonlinear KRR versus linear decoders for test stimuli with different numbers of discs. All decoders were trained only on the 10-disc stimulus. E: Decoding error (MSE; average ± SEM across sites) in fluctuating and constant epochs is significantly larger for linear decoders (p<0.001) relative to nonlinear KRR and the neural network.

Fig 3. Spike-history dependencies affect decoding performance.

A: Shuffles of responses to repeated stimulus presentations remove different types of correlations, but preserve average locking to the stimulus (PSTH), and thus stimulus-induced correlations. B: A repeated stimulus fragment (red trace), nonlinear kernelized decoder predictions using real responses (green), and using responses without different types of correlations (gray); shown is the prediction mean ± SD over repeats. C: Increase in decoding error (MSE) when spike-history dependencies or noise correlations are removed (average ± SEM across sites); percentages report fractional differences relative to the original performance. D: Spike count distributions for a single example cell. Removing spike-history dependencies broadens the distributions, in particular in constant epochs. Dashed line = expectation for a fully randomized spike train with a matched firing rate. E: Variance-to-mean ratio F of spike count distributions for spike trains with and without spike-history dependencies. Each point is a cell that contributes most to decoding at a particular site (when the same cell contributes to multiple sites, average ± SD across sites is shown).

Fig 4. Spike-history dependencies of intermediate strength facilitate nonlinear decoding in simple models of neural processing.

A: Schematic of a single-cell Generalized Linear Model (see Methods). The neuron’s sensitivity to the stimulus is determined by a radially symmetric difference-of-Gaussians spatial filter that has a monophasic timecourse ($\overrightarrow{k}$), and combines additively with the neuron’s sensitivity to its own past spiking, given by filter $\overrightarrow{h}$ (with strong refractoriness followed by weak facilitation). Importantly, $\overrightarrow{h}$ shapes spike-history dependencies in the resulting spike trains. A nonlinear function f(⋅) (here, threshold-linear) of the combined sensitivities gives the neuron’s instantaneous firing rate that can be used to generate individual spike train instances. Shapes, as well as the temporal and spatial scales of the filters, were realistic for our data. B: Example rasters (50 repeats) generated with the encoding model for a given intensity trace and different magnitudes (α) of spiking history filter $\overrightarrow{h}$. The rasters are matched in PSTH (bottom) but differ in temporal noise correlations. C: Average spike count variance-to-mean ratio, F, (± SD) of the model as a function of α in fluctuating and constant epochs. D: Decoding error as a function of α. Decoders are trained for each separate α and tested on withheld stimuli; shade = SD over 10 spike train realizations.