Boost event-driven tactile learning with location spiking neurons

Abstract

Tactile sensing is essential for a variety of daily tasks. Inspired by the event-driven nature and sparse spiking communication of the biological systems, recent advances in event-driven tactile sensors and Spiking Neural Networks (SNNs) spur the research in related fields. However, SNN-enabled event-driven tactile learning is still in its infancy due to the limited representation abilities of existing spiking neurons and high spatio-temporal complexity in the event-driven tactile data. In this paper, to improve the representation capability of existing spiking neurons, we propose a novel neuron model called "location spiking neuron," which enables us to extract features of event-based data in a novel way. Specifically, based on the classical Time Spike Response Model (TSRM), we develop the Location Spike Response Model (LSRM). In addition, based on the most commonly-used Time Leaky Integrate-and-Fire (TLIF) model, we develop the Location Leaky Integrate-and-Fire (LLIF) model. Moreover, to demonstrate the representation effectiveness of our proposed neurons and capture the complex spatio-temporal dependencies in the event-driven tactile data, we exploit the location spiking neurons to propose two hybrid models for event-driven tactile learning. Specifically, the first hybrid model combines a fully-connected SNN with TSRM neurons and a fully-connected SNN with LSRM neurons. And the second hybrid model fuses the spatial spiking graph neural network with TLIF neurons and the temporal spiking graph neural network with LLIF neurons. Extensive experiments demonstrate the significant improvements of our models over the state-of-the-art methods on event-driven tactile learning, including event-driven tactile object recognition and event-driven slip detection. Moreover, compared to the counterpart artificial neural networks (ANNs), our SNN models are 10× to 100× energy-efficient, which shows the superior energy efficiency of our models and may bring new opportunities to the spike-based learning community and neuromorphic engineering. Finally, we thoroughly examine the advantages and limitations of various spiking neurons and discuss the broad applicability and potential impact of this work on other spike-based learning applications.

Keywords: Spiking Neural Networks; event-driven tactile learning; event-driven tactile object recognition; event-driven tactile slip detection; location spiking neurons; robotic manipulation; spiking neuron models.

Figure 1

Recurrent neuronal dynamic mechanisms for the existing spiking neurons of ν = t and location spiking neurons of ν = l. Unlike existing spiking neuron models that update their membrane potentials based on time steps ν = t, location spiking neurons update their membrane potentials based on locations ν = l. (A) The refractory dynamics of a TSRM neuron i or an LSRM neuron i. Immediately after firing an output spike at ${\nu }_i^{(f)}$, the value of u_i(ν) is lowered or reset by adding a negative contribution η_i(·). The kernel η_i(·) vanishes for $\nu <{\nu }_i^{(f)}$ and decays to zero for ν → ∞. (B) The incoming spike dynamics of a TSRM neuron i or an LSRM neuron i. A presynaptic spike at ${\nu }_j^{(f)}$ increases the value of u_i(ν) for $\nu \ge {\nu }_j^{(f)}$ by an amount of $w_{ij}{x}_j({\nu }_j^{(f)}){ϵ}_{ij}(\nu -{\nu }_j^{(f)})$. The kernel ϵ_ij(·) vanishes for $\nu <{\nu }_j^{(f)}$. “ < ” and “≥” indicate the location order when ν = l. (C) The recurrent neuronal dynamics of a TLIF neuron i or an LLIF neuron i. The neuron i takes as input binary spikes and outputs binary spikes. x_j represents the input signal to the neuron i from neuron j, u_i is the neuron's membrane potential, and o_i is the neuron's output. An output spike will be emitted from the neuron when its membrane potential surpasses the firing threshold u_th, after which the membrane potential will be reset to u_reset. This figure is adapted from Kang et al. (2022).

Figure 2

The network structure of the Hybrid_SRM_FC. (The Upper Panel) The SNN with TSRM neurons processes the input spikes X_in and adopts the temporal recurrent neuronal dynamics (shown with red dashed arrows) of TSRM neurons to extract features from the data, where SFc is the spiking fully-connected layer with TSRM neurons. (The Lower Panel) The SNN with LSRM neurons processes the transposed input spikes $X_{in}^{\prime }$ and employs the spatial recurrent neuronal dynamics (shown with purple dashed arrows) of LSRM neurons to extract features from the data, where SFc-location is the spiking fully-connected layer with LSRM neurons. Finally, the spiking representations from two networks are concatenated to yield the final predicted label. (32) and (20) represent the sizes of fully-connected layers, where we assume the number of classes (K) is 20. This figure is adapted from Kang et al. (2022).

Figure 3

(A) The tactile spatial graph G_s at time step t generated by the Minimum Spanning Tree (MST) algorithm (Gu et al., 2020). Each circle represents a taxel of NeuTouch. (B) Based on event sequences, we propose two different tactile temporal graphs G_t for a specific taxel n = 1: the above one is the sparse tactile temporal graph, while the below one is the dense tactile temporal graph.

Figure 4

The structure of the Hybrid_LIF_GNN, where “SSG” is the spatial spiking graph layer, “SSFC” is the spatial spiking fully-connected layer, “TSG” is the temporal spiking graph layer, and “TSFC” is the temporal spiking fully-connected layer. The spatial spiking graph neural network processes the T tactile spatial graphs and adopts the temporal recurrent neuronal dynamics (shown with red arrows) of TLIF neurons to extract features. The temporal spiking graph neural network processes the N tactile temporal graphs and employs the spatial recurrent neuronal dynamics (shown with purple arrows) of LLIF neurons to extract features. Finally, the model fuses the predictions from two networks and obtains the final predicted label. (3, 64) represents the hop size and the filter size of spiking graph layers. (128), (256), and (10) represent the sizes of fully-connected layers, where we assume the number of classes (K) is 10.

Figure 5

Location orders. (A) Arch-like location order. (B) Whorl-like location order. (C) Loop-like location order. (D) Random location order.

Figure 6

The timestep-wise inference (Algorithm 1) for the SNN with TSRM neurons (SNN_TSRM), the SNN with LSRM neurons (SNN_LSRM), the Hybrid_SRM_FC, and the time-weighted Hybrid_SRM_FC on (A) “Objects-v1,” (B) “Slip Detection,” and (C) “Containers-v1.” Please note that we use the same event sequences as Taunyazoz et al. (2020) and the first spike occurs at around 2.0 s for “Objects-v1” and “Containers-v1.” From the figure, we can see that the models with location spiking neurons have not reached the saturated levels while the blue line (the models with only traditional spiking neurons) has already reached the saturated levels. This demonstrates the potential of location spiking neurons and the models with location spiking neurons could provide the better performance by increasing the time on these tasks.

Figure 7

The timestep-wise inference (Algorithm 2) accuracies (%) for the spatial spiking graph neural network (SSGNN), the temporal spiking graph neural network (TSGNN), the Hybrid_LIF_GNN, and the time-weighted Hybrid_LIF_GNN on (A) “Objects-v0” and (B) “Containers-v0".

Figure 8

The Hybrid_SRM_FC processes a spike audio sequence and predict its label. The network structure of this model is the same as what we show in Figure 2.