Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

Abstract

A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possess close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov & Hopfield (2020) to express general associative memory models using neural network dynamics with local computation, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.

Figure 1

Left: Schematic of the key equations that make up the general theory of the abstract Hopfield network, which shows the factorization of a UHN into similarity, separation, and projection. Right: Visual representation of the factorization diagram when performing an associative memory task on three stored memories. The corrupted data point is scored against the three memories (similarity). The difference in scores are then exaggerated (separation), and used to retrieve a stored memory (projection).

Figure 2. Capacity of the associative memory networks with different similarity functions, as measured by increasing the number of stored images.

The capacity is measured as the fraction of correct retrievals. To test retrieval, the top-half of the image was masked with all zeros (this is equivalent to a fraction masked of 0.5 in Figure 4) and was then presented as the query vector for the network. A retrieval was determined to be correct if the summed squared difference between all pixels in the retrieved image and the true reconstruction was less than a threshold T, which was set at 50. The queries were presented as the stored images corrupted with independent Gaussian noise with a variance of 0.5. Mean retrievals over 10 runs with different sets of memories images. Error bars are computed as the standard deviations of the correct retrievals of the 10 runs. A softmax separation function was used with a β parameter of 100.

Figure 3. The retrieval capacity of the network on retrieving half-masked images using the dot-product similarity function.

Plotted are the means and standard deviations of 10 runs. A query was classed as correctly retrieved if the sum of squared pixel differences was less than a threshold of 50.

Figure 4

Top Row: Retrieval capability against increasing levels of i.i.d added to the query images for different similarity functions. Bottom Row: Retrieval capability against increasing fractions of zero-masking of the query image. The networks used a memory of 100 images with the softmax separation function. Error bars are across 10 separate runs with different sets of memories stored. Datasets used left to right: MNIST, CIFAR, and Tiny ImageNet.