Decoding the brain: From neural representations to mechanistic models

Abstract

A central principle in neuroscience is that neurons within the brain act in concert to produce perception, cognition, and adaptive behavior. Neurons are organized into specialized brain areas, dedicated to different functions to varying extents, and their function relies on distributed circuits to continuously encode relevant environmental and body-state features, enabling other areas to decode (interpret) these representations for computing meaningful decisions and executing precise movements. Thus, the distributed brain can be thought of as a series of computations that act to encode and decode information. In this perspective, we detail important concepts of neural encoding and decoding and highlight the mathematical tools used to measure them, including deep learning methods. We provide case studies where decoding concepts enable foundational and translational science in motor, visual, and language processing.

Keywords: BCIs; data-driven; decoding; deep learning; encoding; language; normative models.

Figure 1.. Encoding-Decoding across scales.

A: An encoder represents the neural response of population K(t) to stimulus x(t) via P(K|x), and a decoder aims to recover x(t) given the neural activity K(t) via P(x|K). B: Systems neuroscience spans scales of descriptions and decoding algorithms can target any individual level, and even span across scales. Here, we outline example scales (from genes to environment), the types of data we can collect (from genetic sequencing to whole-animal video analysis), and the classes of models the field has developed. On the far right is our mapping of scales, example data, and example tools to levels of understanding. Inset images adapted from: (–8).

Figure 2.. Encoding model and decoding methods.

A: The cercal system of the cricket has four interneurons that represent the wind direction. The preferred wind directions of the neurons are pointing in four cardinal directions and can be represented by orthogonal vectors (on the left). Each neuron responds with a firing rate approximated by a half-wave rectified cosine function. The maximum firing rate is elicited when the wind is blowing in the preferred direction. B: The wind direction x can be decoded as the direction of the population vector $\widehat{x}$. This vector is the sum of the four preferred orientations scaled by their firing rate. An example is shown for neurons responding with activity [36, 12, 2, 1]^T. Note how the population vector closely matches the wind direction. C: In the k-nearest neighbors (k-NN) decoding method, neural activity K is represented within a neural activity space, which is illustrated here in 2D for two neurons for clarity (neuron 1 and neuron 2 from panel A). With these two neurons, angles between 0° and 225° can be represented. For simplicity, we focus on a nearest neighbor variant with k = 1. As 1-NN is only able to decode discrete variable, we classify the angles in three ranges: 0°−45°, 45°−135° and 135°−225°. Previously observed trials are color-coded by their associated wind direction ranges (L = 13). To decode the wind direction for a new trial (unfilled triangle), the k-nearest neighbors (here, k = 1) in the activity space are identified. The decoded wind direction corresponds to the wind associated with the nearest neighbor, highlighted by the sample connected to the observed sample via a dashed line. D: Bayesian decoders incorporate a prior P(x) (dashed line) that reflects the probability of different wind directions before taking neural evidence into account and influences the decoded angle. For instance, if mainly wind directions around 125° have been experienced (mean of the prior P(x)), the decoded angle will be shifted towards this direction. The likelihood P(K|x) (green-blue line) describes the probability of observing a particular neural response K given a specific wind direction. Following Bayes’ theorem Eq. (3), the prior P(x) and the likelihood P(K|x) are multiplied to obtain the posterior distribution P(x|K) (solid black line). The posterior can be used to decode the wind direction, here 270°, based on highest value for the observed neural activity K.

Figure 3.. Data-driven models: statistical power vs. mechanistic realism.

Ultimately, as a field we want to map mechanism to computation in order to have causal, testable models. On one side we have mechanistic models, and on the other statistical models that aim to best encode neural dynamics, but there is a large gap between them. We provide a non-exhaustive selection of contributions: (, , , –46).

Figure 4.. Learnable latent variable models.

A: On the path to building more causal models are new frameworks, such as CEBRA (13), that allow for learning a mapping from the observable data K to the latent dynamics Z. Here, the aim is to use identifiable models with contrastive learning g(Z) (the encoder), then invert this model or use another decoder framework to probe the relationship between the estimated latents, Z¹, and a variable such as an externally observable state (behavior), internal, or sensory (i.e., recover some stimulus space (x)).

Figure 5.. Examples of decoding from motor, vision, and language areas.

A: Close loop experiments using digital twins: Schematic of an inception loop, depicted clockwise from the upper left: 1) Presentation of large entropy natural stimuli and tasks while recording large-scale neural activity, 2) Deep learning models accurately predict neural activity, creating a functional digital twin of the recorded neurons, 3) The in silico model facilitates unlimited experiments and employs mechanistic interpretability tools to characterize neural tuning, and 4) Images and hypotheses synthesized in silico are validated back in vivo. B: Illustration of decoded images from fMRI using Diffusion Models: Ground truth (GT) vs. decoded images generated by Chen et al. (94) from human fMRI with a diffusion model. Note that decoded images share similar color, shape and semantics. C: Multi-modal speech decoding: Adapted from Metzger et al. (95), this panel shows the decoding pipeline, where neural activity was used to train a ANN to predict phone probabilities, speech-sound features and articulatory gestures. A decoder was then constructed to produce text, generate audible speech and animate an avatar, respectively.