In Search of Dispersed Memories: Generative Diffusion Models Are Associative Memory Networks

Abstract

Uncovering the mechanisms behind long-term memory is one of the most fascinating open problems in neuroscience and artificial intelligence. Artificial associative memory networks have been used to formalize important aspects of biological memory. Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. Similar to associative memory systems, these networks define a dynamical system that converges to a set of target states. In this work, we show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is (asymptotically) identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network. Leveraging this connection, we formulate a generalized framework for understanding the formation of long-term memory, where creative generation and memory recall can be seen as parts of a unified continuum.

Keywords: associative memory networks; generative diffusion models; hopfield networks.

Figure 1

Visualization of different kinds of energy landscape and gradient vector fields corresponding to different forms of memory (in a two-dimensional space): (a) classical point-like memory; (b) extended localized memory; (c) non-localized (semantic) memory structure. The color denotes the probability density of the learned distribution while the lines represent the integral trajectories of the vector field oinduced by the score function.

Figure 2

Qualitative analysis of the (marginal) denoising trajectories of a binary associative memory problem with four patterns in a five-dimensional space. (a) Comparison between denoising trajectories of diffusion models and modern Hopfield updates. The diffusion curves are integrated using the Euler method with 2000 steps. The trajectories are overlaid to four modern Hopfield updates. (b) Comparison between exact and learned deterministic denoising trajectories. The colors are used to identify individual trajectories.

Figure 3

(a) Median error of exact diffusion model as function of the dimensionality. (b) Capacity of diffusion models and Hopfield networks in log scale. The shaded area denotes the estimated $95\%$ intervals.