Characterization of Generalizability of Spike Timing Dependent Plasticity Trained Spiking Neural Networks

Abstract

A Spiking Neural Network (SNN) is trained with Spike Timing Dependent Plasticity (STDP), which is a neuro-inspired unsupervised learning method for various machine learning applications. This paper studies the generalizability properties of the STDP learning processes using the Hausdorff dimension of the trajectories of the learning algorithm. The paper analyzes the effects of STDP learning models and associated hyper-parameters on the generalizability properties of an SNN. The analysis is used to develop a Bayesian optimization approach to optimize the hyper-parameters for an STDP model for improving the generalizability properties of an SNN.

Keywords: Bayesian optimization; Hausdorff dimension; addSTDP; generalization; leaky integrate and fire; logSTDP; multSTDP; spiking neural networks.

Figure 1

(A) Resulting weight distribution for log-STDP (Gilson and Fukai, 2011);multSTDP (Van Rossum et al., 2000) and add-STDP (Song et al., 2000). (B) Plot for Functions a₊ for LTP and − f₋ for LTD in log-STDP (blue solid curve); mult-STDP (orange dashed line); and add-STDP model green dashed-dotted curve for depression and orange dashed curve for potentiation.

Figure 2

Plot showing the trajectories of the α−stable Lévy process $L_t^{\alpha }$ for varying values of α.

Figure 3

Figure showing the probability density functions of the α−stable Lévy process $L_t^{\alpha }$ for varying values of α.

Figure 4

The intensity values of the MNIST image are converted to Poisson-spike trains. The firing rates of the Poisson point process are proportional to the intensity of the corresponding pixel. These spike trains are fed as input in an all-to-all fashion to excitatory neurons. In the figure, the black shaded area from the input to the excitatory layer shows the input connections to one specific excitatory example neuron. The red shaded area denotes all connections from one inhibitory neuron to the excitatory neurons. While the excitatory neurons are connected to inhibitory neurons via one-to-one connection, each of the inhibitory neurons is connected to all excitatory ones, except for the one it receives a connection from.

Figure 5

Neuron connection weights for learned digit representations for (A) SFR = 0.9 and (B) SFR = 2.1.

Figure 6

Figure showing the variation in the training loss with increasing iterations for different types of STDP models keeping (A) SFR= 0.9 and (B) η = 0.05.

Figure 7

Figure showing the change in training loss with iterations for varying scaling function ratios for the log-STDP learning process.

Figure 8

Figure showing the change in training loss with iterations for varying learning rates for the log-STDP learning process.

Figure 9

(A) Plots for the impact of the scaling function ratios on generalization (results shown in Table 2). (B) Plots for the impact of the learning rates on generalization (results shown in Table 3).

Figure 10

Plot showing the change of BG Index, Training and Testing Accuracy between the add-STDP and log-STDP over functional evaluations during Bayesian Optimization.