Spiking Autoencoders With Temporal Coding

Abstract

Spiking neural networks with temporal coding schemes process information based on the relative timing of neuronal spikes. In supervised learning tasks, temporal coding allows learning through backpropagation with exact derivatives, and achieves accuracies on par with conventional artificial neural networks. Here we introduce spiking autoencoders with temporal coding and pulses, trained using backpropagation to store and reconstruct images with high fidelity from compact representations. We show that spiking autoencoders with a single layer are able to effectively represent and reconstruct images from the neuromorphically-encoded MNIST and FMNIST datasets. We explore the effect of different spike time target latencies, data noise levels and embedding sizes, as well as the classification performance from the embeddings. The spiking autoencoders achieve results similar to or better than conventional non-spiking autoencoders. We find that inhibition is essential in the functioning of the spiking autoencoders, particularly when the input needs to be memorised for a longer time before the expected output spike times. To reconstruct images with a high target latency, the network learns to accumulate negative evidence and to use the pulses as excitatory triggers for producing the output spikes at the required times. Our results highlight the potential of spiking autoencoders as building blocks for more complex biologically-inspired architectures. We also provide open-source code for the model.

Keywords: autoencoders; backpropagation; biologically-inspired artificial intelligence; inhibition; latency coding; spiking networks; temporal coding.

Figure 1

Illustration of membrane potential dynamics for a neuron with θ = 0.5 and τ = 1. The neuron receives input spikes at times t_i ∈ {1, 4, 5, 8, 12, 17, 19} with corresponding weights w_i ∈ {0.5, 0.3, 0.4, −0.2, −0.3, 1.2, 0.9}, which cause it to spike at t_out = 19.39.

Figure 2

Architecture of the spiking autoencoder. The weights and the pulses are trainable.

Figure 3

Reconstruction errors for spiking (“snn”) and non-spiking (“ann”) autoencoders at different levels of noise, for embedding sizes 8, 16, and 32, on the MNIST and FMNIST datasets.

Figure 4

A digit from the MNIST test set reconstructed by a spiking autoencoder with embedding size 32 and target latency l = 1, at different levels of noise.

Figure 5

Visualisation of MNIST embeddings produced by a spiking autoencoders with target latency l = 1 at different levels of noise η and embedding sizes h, using the t-distributed stochastic neighbour embedding (t-SNE) technique, with perplexity set to 20. The results are qualitatively similar for different perplexity values. Axis units (not shown) are arbitrary and identical for each plot.

Figure 6

Interpolating between four items from the MNIST and FMNIST test sets in embedding space. The embeddings are generated by a spiking autoencoder with hidden layer size 32, target latency l = 1, noise level η = 0. They are then interpolated and, finally, run through the decoder layer to obtain the representation in original space.

Figure 7

Accuracy of an SVM classifying embeddings produced by spiking (“snn”) and non-spiking (“ann”) autoencoders at different levels of noise, for embedding sizes 8, 16, and 32, on the MNIST dataset. The baseline is the classification accuracy on the original set.

Figure 8

Spike distributions on the full test set in trained spiking autoencoders with embedding size 32, noise level η = 0, target latencies l = 1 and l = 16. The pulses are shown individually.

Figure 9

Output potentials during the reconstruction of a test example by spiking autoencoders with embedding size 32, noise level η = 0, target latencies l = 1 and l = 16. The output neuron is chosen such that the target spike time is smaller than l + 0.1 (in other words, it is located in the centre of the image and encodes salient digit information). The figure underlines the initial negative response of the membrane voltage, followed by a positive response caused by pulses.

Figure 10

Weight distributions in spiking autoencoders, for regular neurons and pulses. All models have embedding size h = 32 and noise level η = 0.

Figure 11

Reconstruction loss on the inverted-brightness MNIST dataset for spiking (“snn”) and non-spiking (“ann”) autoencoders. The embedding size is always h = 32. The spiking autoencoder has a target latency of l = 1. The non-spiking networks have either ReLU activation functions in the encoder and sigmoid activation functions in the decoder, or zero-centred Gaussian-like activation functions everywhere.