Information Thermodynamics: From Physics to Neuroscience

Abstract

This paper provides a perspective on applying the concepts of information thermodynamics, developed recently in non-equilibrium statistical physics, to problems in theoretical neuroscience. Historically, information and energy in neuroscience have been treated separately, in contrast to physics approaches, where the relationship of entropy production with heat is a central idea. It is argued here that also in neural systems, information and energy can be considered within the same theoretical framework. Starting from basic ideas of thermodynamics and information theory on a classic Brownian particle, it is shown how noisy neural networks can infer its probabilistic motion. The decoding of the particle motion by neurons is performed with some accuracy, and it has some energy cost, and both can be determined using information thermodynamics. In a similar fashion, we also discuss how neural networks in the brain can learn the particle velocity and maintain that information in the weights of plastic synapses from a physical point of view. Generally, it is shown how the framework of stochastic and information thermodynamics can be used practically to study neural inference, learning, and information storing.

Keywords: computational neuroscience; inference; information; learning; neurons and synapses; non-equilibrium stochastic thermodynamics; plasticity.

Figure 1

Stimulus-induced transition from weak to strong synapses. Transient input $c\left(v\right)$ to the neuron can induce a transition in the collective weight of synapses $\overline{w}$ (upper panel). Transitions from weak ($\overline{w}\approx {\overline{w}}_d$) to strong ($\overline{w}\approx {\overline{w}}_u$) synapses take place only when the amplitude of synaptic plasticity $\lambda$ or firing rate of presynaptic neurons $\overline{f}$ are sufficiently large (middle and lower panels). Note that $\overline{w}$ can maintain the value ${\overline{w}}_u$ for a very long time, much larger than the synaptic time constant ${\tau }_w=200$ s (synaptic memory trace about c), because collective stochastic fluctuations are rescaled by the number of synapses $1/\sqrt{N_s}$. The middle and lower panels look almost identical despite different parameters, because the noise term in Equation (60) dominates for most of the time in this regime. The nominal parameters used are $\lambda =1.3$, $\beta =1.2$, $\overline{f}=0.9$ Hz, ${\tau }_n=0.3$ s, ${\tau }_w=200$ s, ${\sigma }_w=5.0$, $N_s=1000$, $r_m=10$ Hz, $u=10$ mm/s, $ϵ=0.1$ mm/s. In this example, the stimulus moves with the linearly increasing velocity $v=0.02t+7$ (mm/s) with a small accelaration of $0.02$ mm/s². Too large accelaration prohibits the synaptic transition to the state with ${\overline{w}}_u$.

Figure 2

Effective potential $V(\overline{w},c)$ for the collective synaptic weights and bistability. (A) The core potential $V_0\left(\overline{w}\right)$ has either one minimum, for sufficiently weak plasticity amplitude $\lambda$, or two minima for stronger $\lambda$. The latter corresponds to bistability in the collective behavior of synapses. Note that the miniumum at $\overline{w}=0$ is very shallow (inset). (B) The bistability regime. The presence of even a weak stimulus $c\left(v\right)$ lowers the potential barrier in $V(\overline{w},c)$ between the shallow and the deep minima, which can facilitate a transition from weak to strong synapses ($\overline{w}$ can change from ${\overline{w}}_d$ to ${\overline{w}}_u$). The parameters used are the same as in Figure 1.