DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

Abstract

An increasing number of studies are highlighting the importance of spatial dendritic branching in pyramidal neurons in the neocortex for supporting non-linear computation through localized synaptic integration. In particular, dendritic branches play a key role in temporal signal processing and feature detection. This is accomplished thanks to coincidence detection (CD) mechanisms enabled by the presence of synaptic delays that align temporally disparate inputs for effective integration. Computational studies on spiking neural networks further highlight the significance of delays for achieving spatio-temporal pattern recognition with pure feed-forward neural networks, without the need of resorting to recurrent architectures. In this work, we present "DenRAM", the first realization of a feed-forward spiking neural network with dendritic compartments, implemented using analog electronic circuits integrated into a 130 nm technology node and coupled with Resistive Random Access Memory (RRAM) technology. DenRAM's dendritic circuits use RRAM devices to implement both delays and synaptic weights in the network. By configuring the RRAM devices to reproduce bio-realistic timescales, and by exploiting their heterogeneity we experimentally demonstrate DenRAM's ability to replicate synaptic delay profiles, and to efficiently implement CD for spatio-temporal pattern recognition. To validate the architecture, we conduct comprehensive system-level simulations on two representative temporal benchmarks, demonstrating DenRAM's resilience to analog hardware noise, and its superior accuracy compared to recurrent architectures with an equivalent number of parameters. DenRAM not only brings rich temporal processing capabilities to neuromorphic architectures, but also reduces the memory footprint of edge devices, warrants high accuracy on temporal benchmarks, and represents a significant step-forward in low-power real-time signal processing technologies.

Fig. 1. Dendritic RRAM (DenRAM) concept.

a Depiction of a biological Neuron, receiving input spikes through multiple dendritic branches. The packet of neurotransmitters travels across the dendritic branch before reaching the neuron’s soma, where it is integrated. b Scheme of the Dendritic Network, formed by several Dendritic circuits grouped into Dendritic branches macro-circuits, highlighted by different colors. The branches' outputs are integrated into a Leaky-Integrate-and-Fire Neuron. c State-of-the-art results on the SHD dataset as a function of the number of parameters. Delay-based networks show higher accuracy and lower memory footprint compared to recurrent architectures (SRNN: recurrent spiking neural networks, A-SRNN: augmented-SRNN). d Recurrent Neural Networks are hard to train and yield low performance. Dendritic SNNs are feed-forward models that perform better than RNNs despite reduced Memory Footprint and Power Consumption. e Applications for the Dendritic SNN include Key-Word-Spotting and Heartbeat anomaly detection, and possibly many other sequence processing tasks. Illustrations in a, d, and e were created with Inkscape.

Fig. 2. Dendritic circuit, the building block of the DenRAM architecture.

a Detailed schematics of the Dendritic circuit, featuring the Delay and Weight RRAM devices, a Capacitor, dedicated multiplexers (MUX) for switching between programming and reading operations, and a Threshold circuit. b Scanning Electron-Microscopy image of a Hf_xO-based RRAM device used in the Dendritic circuit. c Measurement of the Dendritic Circuit, featuring the voltage on the Capacitor (V_cap), and output (V_OUT). The input voltage pulse IN is applied at t = 0 s and is not shown in the plot. d Probability Distribution Function (PDF) of the delay measurements, with a log-normal distribution fitting curve. e Effect of the Weight RRAM on the output current I_OUT measured from the Dendritic Circuit. Higher values of conductance (conductance G₈ larger than G₄, referencing the conductance levels in f) increase the output current I_OUT. f Cumulative Distribution Function (CDF) of the Weight RRAM conductance values measured in a 16kb array, in different resistive states. g Breakdown of the dynamic power consumption of the dendritic circuit, showing the contributions from all the components in part (a). The highest power is attributed to the Threshold block responsible for the 66.7% of the total consumption.

Fig. 3. Coincidence Detection with Dendritic Circuits.

a Schematics of the DenRAM architecture. Two input spikes are processed by different dendritic branches, each with different set of delays. The delays that align the two input spikes receive large weights. The inputs are broadcasted to different dendritic trees leading to different output neurons. b CD mechanism on a biological neuron, where two inputs are fed from two different synapses and reach the soma at the same time, thanks to dendritic delays. c Physical layout of DenRAM. d Measurement of the Dendritic Network performing coincidence detection. Illustration in (b) created with Inkscape.

Fig. 4. Performance of DenRAM on Heartbeat Anomaly Detection and Keyword Spotting.

a Classification accuracy as a function of the number of synapses per dendritic branch and comparison with a SRNN of 32 neurons (1.1k parameters). Error bars capture the standard deviation over 10 trials. b Memory Footprint of the DenRAM architecture solving ECG, compared with an iso-accuracy SRNN. c Power consumption of DenRAM in the ECG task and comparison an iso-accuracy SRNN. d Classification accuracy as a function of the noise introduced on the weights for two delay architectures (D1: 700 inputs, 16 delays; D2: 256 inputs, 16 delays) and for two SRNN architectures with one hidden layer (R1: 700 inputs, 235 hidden neurons; R2: 256 inputs, 180 hidden neurons). e Classification accuracy (with RRAM-calibrated noise on the weights) concerning the network’s number of parameters for D1, D2 and R2, sweeping the number of synapses per branch for D1 and D2, the number of hidden neurons for R2. f Power consumption of each network configuration (D1, D2 and R2) shown in e). In d–f error bars represent the standard deviation over 3 trials.