Tactile sensory coding and learning with bio-inspired optoelectronic spiking afferent nerves

Abstract

The integration and cooperation of mechanoreceptors, neurons and synapses in somatosensory systems enable humans to efficiently sense and process tactile information. Inspired by biological somatosensory systems, we report an optoelectronic spiking afferent nerve with neural coding, perceptual learning and memorizing capabilities to mimic tactile sensing and processing. Our system senses pressure by MXene-based sensors, converts pressure information to light pulses by coupling light-emitting diodes to analog-to-digital circuits, then integrates light pulses using a synaptic photomemristor. With neural coding, our spiking nerve is capable of not only detecting simultaneous pressure inputs, but also recognizing Morse code, braille, and object movement. Furthermore, with dimensionality-reduced feature extraction and learning, our system can recognize and memorize handwritten alphabets and words, providing a promising approach towards e-skin, neurorobotics and human-machine interaction technologies.

Fig. 1. Schematic diagram of the biological and artificial afferent nerve systems.

a In the biological afferent nerve, external pressures applied to the skin change the potentials of receptors that are embedded in the skin. The cell body of the sensory neuron integrates the potentials and initiates action potentials (spikes) with coded pressure information. The axon transmits the action potentials to the axon terminals, which form synapses with interneurons, where they induce post-synaptic currents (PSCs). The central nervous system (CNS) processes the pressure information by integrating the action potentials from multiple synapses. b In the artificial afferent nerve, external pressures applied to the e-skin change the resistance of MXene in the flexible pressure sensor. The ADC-LED circuit, consisting of a ring oscillator, an edge detector and an LED, receives the voltage signal from the MXene sensor and initiates optical spikes with coded pressure information. The optical spikes are transmitted to a synaptic photomemristor (OE synapse), which integrates and processes the spikes into a PSC to decode and memorize the pressure information.

Fig. 2. Characterization of the optoelectronic spiking afferent nerve.

a I–V curves of the MXene-based pressure sensor with applied pressures from 0 to 200 kPa. b Resistance and resistance change ratio in response to an increasing pressure. c Output frequency and amplitude of the pressure-dependent ADC for an increasing pressure up to 100 kPa. The insert shows the analog-to-digital conversion of electrical signals in the ADC. d I–t curve of the synaptic photomemristor with optical pulses as stimuli, showing current spikes and persistent photoconductivity (PPC) in response to the optical pulses. e Optical paired pulse facilitation (PPF) or neural facilitation behavior with respect to current spiking induced by optical spiking. f Spike-rate-dependent plasticity (SRDP) behavior with respect to PPC induced by optical spiking. g Input–output of the system showing the correlation among pressures, ADC outputs, and post-synaptic currents (PSCs). With increasing pressure, both the frequency of the PSC (h) and the weight change Δw (i) increase. The error bars indicate variations during repeated measurements.

Fig. 3. Pressures integration and motion detection.

a Schematic diagram of an optoelectronic spiking afferent nerve with two branches. b PSC measured with only one pressure 35 kPa (first panel) or 90 kPa (second panel) applied to one of the sensors. PSC measured with the same pressures applied to the two sensors simultaneously (third panel) and the numerical sum of the two PSCs shown in the first and second panel (fourth panel). c Fourier transformed spectra of the four signals shown in b. The peaks at 59 Hz and 80 Hz contain information on the pressure amplitude. One synaptic photomemristor can combine and integrate multiple pressures. d Schematic diagram of a 2 × 2 optoelectronic spiking afferent nerve for motion detection. e, f PSCs and frequencies detected when touching the sensor array from 1a to 2a. g, h PSCs and frequencies detected when moving a finger from sensor 1a to 2b. i, j PSCs and frequencies detected when touching the sensor array from 1a to 1b. k Average speed of touch motion in the three cases. l Image of a flexible 4 × 4 sensor array. The scale bar corresponds to 1 cm. m Detected PSCs from the sensor array when moving a finger over the array in circular motion. n Motion path and spiking frequency containing information on the pressure amplitude extracted from the PSCs in m.

Fig. 4. Recognition of handwritten letters of the alphabet with dimensionality-reduced features.

a Schematic diagram of handwriting recognition with feature extraction and feature learning using automatic hardware-based dimensionality reduction. b Structure and working principle of the optoelectronic spiking neural network. c Measured PSCs of five photomemristors with a handwritten ‘A’ as input. d Spiking proportions $\overrightarrow{P_A}$ extracted from c. The insert shows the relative activities of the five photomemristors with $\overrightarrow{P_A}$ in radar chart representation. e Learned feature dictionary of handwritten letters of the alphabet from the first input epoch. f Recognition accuracy of handwritten letters based on the learned feature dictionary. g Weight change evolution of the photomemristors during training and testing with a handwritten ‘A’ as input. h Weight change after 20 cycles for handwritten letters of the alphabet, demonstrating learning and memory capabilities.

Fig. 5. Classification of handwritten words with enhanced dimensionality reduction.

a Schematic diagram of the handwritten word ‘APPLE’. b Dimensionality reduction of the vector representing ‘APPLE’ from 25 (five per letter) to 15 by combining ‘A’ and ‘P’, ‘P’, and ‘L’. c Schematic diagram of the artificial neural network that processes the dimensionality-reduced vectors. The 15 vector elements are used as inputs and the six output neurons represent the six handwritten words. d–i Training results for the recognition of d ‘APPLE’, e ‘ORANGE’, f ‘BANANA’, g ‘PEAR’, h ‘CHERRY’, and i ‘GRAPE’ during the first 10 epochs. j Response of the six output neurons upon handwriting of the fruit names after the neural network is trained.