Bio-Inspired Neuromorphic Sensory Systems from Intelligent Perception to Nervetronics

Abstract

Inspired by the extensive signal processing capabilities of the human nervous system, neuromorphic artificial sensory systems have emerged as a pivotal technology in advancing brain-like computing for applications in humanoid robotics, prosthetics, and wearable technologies. These systems mimic the functionalities of the central and peripheral nervous systems through the integration of sensory synaptic devices and neural network algorithms, enabling external stimuli to be converted into actionable electrical signals. This review delves into the intricate relationship between synaptic device technologies and neural network processing algorithms, highlighting their mutual influence on artificial intelligence capabilities. This study explores the latest advancements in artificial synaptic properties triggered by various stimuli, including optical, auditory, mechanical, and chemical inputs, and their subsequent processing through artificial neural networks for applications in image recognition and multimodal pattern recognition. The discussion extends to the emulation of biological perception via artificial synapses and concludes with future perspectives and challenges in neuromorphic system development, emphasizing the need for a deeper understanding of neural network processing to innovate and refine these complex systems.

Keywords: artificial synapse; bio‐inspired; machine learning; neuromorphic sensory system.

Figure 1

An illustration depicting a visual representation of both human and artificial signal processing pathways from various sensory inputs. The human nervous system consists of receptors, sensory nerves, and neurons for sensing, transmitting, and processing of sensory information (left part). Similarly, neuromorphic artificial sensory system mirroring the human nervous system, comprises sensing, transmission, and neural network algorithms for detecting and processing of sensory data (right part).

Figure 2

Artificial vision sensory systems based on synaptic devices a) Schematic illustration of visual signal transmission of the biological sensory system and artificial photonic synapse. b) Illustration of the simulated FONN for image recognition by the photonic synapse. c) Illumination intensities of the OPTs‐based tetrachromatic vision system. d) Detection and memorization of the letter T under UV light stimulation. e) Illustration of ANN structure for digit recognition. f) Comparing recognition accuracy with and without pre‐processing features. g) Schematic illustration of the h‐BN/WSe₂ device integrated with a photodetector mirroring the human optic nerve system. h,i) Recognition of 28 × 28 RGB‐colored images based on the artificial ONN. j) Recognition rate as a function of training epochs with and without ONN. k) Weight mapping images after the 12th and 600th training epochs. (a,b) Reproduced with permission.^{[ 58 ]} Copyright 2022, Springer Nature. (c‐f) Reproduced with permission.^{[ 47 ]} Copyright 2023, Springer Nature. (g‐k) Reproduced with permission.^{[ 54 ]} Copyright 2018, Springer Nature.

Figure 3

Artificial tactile sensory systems based on synaptic devices a) Configuration of artificial sensory nerve comprising of pressure sensors, ring oscillator, and synaptic transistor in comparison with a biological sensory system. b) Illustration of the writing of English letters by the sensory system on a “pen”, and the processing of collected data by the KNN algorithm to identify different letters. c) Recognition accuracy of each English letter. d) Illustration of the structure and working mechanism of the optoelectronic SNN. e) Weight changes depicting the learning and memory capabilities for the handwritten letters of the alphabet. f,g) Dimensionality reduction of the vector representing “APPLE” using artificial neural network and the training results depicting the recognition of various words. h) OECT‐based artificial tactile system that integrates tactile perception, both STP and STP, and learning functionalities within a single device. i) Ion trap and release in NeuroMAT and tactile perception performance under applied tactile stimulation. (a) Reproduced with permission.^{[ 41 ]} Copyright 2018, American Association for the Advancement of Science. (b,c) Reproduced with permission.^{[ 48 ]} Copyright 2019, Wiley‐VCH. (d‐g) Reproduced with permission.^{[ 46 ]} Copyright 2020, Springer Nature. (h) Reproduced with permission from.^{[ 66 ]} Copyright 2020 American Chemical Society. (i) Reproduced with permission.^{[ 68 ]} Copyright 2023, American Association for the Advancement of Science.

Figure 4

Artificial auditory sensory systems based on synaptic devices a) Auditory signal transmission of the biological sensory system. b) Operation mechanism of TENG actuated artificial auditory system. c) Distinguishing different words based on EPSC amplitude. d,e) Acoustic signal recognition by KNN algorithm. f,g) Detection and recognition of sound location using an ANN algorithm. h) Spiking frequency dependency on sound pressure level. i,j) Musical pitch classification through SNN algorithm with single‐layer perceptron. (a) Reproduced with permission.^{[ 70 ]} Copyright 2023, Elsevier Ltd. (b‐g) Reproduced with permission.^{[ 26 ]} Copyright 2020, Elsevier Ltd. (h‐j) Reproduced with permission.^{[ 70 ]} Copyright 2023, Elsevier Ltd.

Figure 5

Artificial chemical sensory systems based on synaptic devices a) Schematics of biological olfactory and gustatory system and signal transmission pathways. b) LTP/LTD characteristics by NO₂ concentrations (5 and 40 ppm). c,d) Circuit diagram of feedback‐controlled response system and operation characteristics under NO₂ environment. e) Schematic illustration of artificial neuron module device consisting of SMO gas sensor and a single transistor neuron. f,g) Classification of four gas types (acetones, ammonia, carbon monoxide, NO₂) by SNN algorithm. h) Structure of MOSFET‐based artificial gustatory neuron. i) Identification of two liquids (Vinegar and Brine) by comparing spiking frequency of synaptic currents. j) EPSC under different concentrations of salt solution. k) Excessive‐intake warning system utilizing light indicator activated by salt concentration conditions. (a) Reproduced with permission.^{[ 89 ]} Copyright 2016, Springer Nature; Reproduced with permission.^{[ 107 ]} Copyright 2022, American Chemical Society. (b‐d) Reproduced with permission.^{[ 105 ]} Copyright 2022, Wiley‐VCH. (e‐g) Reproduced with permission.^{[ 28 ]} Copyright 2022, Wiley‐VCH. (h,i) Reproduced with permission.^{[ 114 ]} Copyright 2022, American Chemical Society. (j,k) Reproduced with permission.^{[ 107 ]} Copyright 2022, American Chemical Society.

Figure 6

Multimodal NAS systems a) Configuration of bimodal sensory system with visual‐haptic (VH) fusion b) Multi‐transparency pattern recognition based on unimodal information (visual or haptic) and VH fusion information. c) Recognition accuracy of results based on unimodal and VH fusion information. d) Illustration of the structure and operation of the artificial MSeNN. e) Schematics of tactile input recognition, visualization, and memorization. f) Images, vision memory, and recognition of handwritten alphabet letters A‐M h) Recognized and reproduced images, smell and tastes of apple, pear, and blueberry upon associated audio input. i) Schematic illustration of the self‐powered VTT. j) Output voltage of the TENG as a function of distance. k) Recognition of six emotions (anger, fear, disgust, happiness, surprise, and sadness) by the multimodal emotion recognition system. l) Comparing emotion recognition accuracy at different input models. (a‐c) Reproduced with permission.^{[ 116 ]} Copyright 2020, Springer Nature. (d‐h) Reproduced with permission.^{[ 49 ]} Copyright 2021, Springer Nature. (i‐l) Reproduced with permission.^{[ 118 ]} Copyright 2022, Springer Nature.

Figure 7

Application of NAS systems in sensory robotics a) Configuration of light‐sensitive sensorimotor nervetronics and photographs of the polymer actuator according to the presynaptic spikes with 0% or 100% strain. b) Schematics and signal flow of ocular prosthesis system. c) Circuit diagram and the photographs of pupillary light reflex and corneal reflex in operation. d) Illustration of the threshold of the computational e‐skin with a teacher signal and the firing pattern of the second‐order neuron before and after associative learning, under an applied force. e) Images depicting the acquired pain reflex after associative learning. f) Circuit diagram of the NeuroMAT‐based artificial tactile memory neuron. g) Photographs showing the variations in pulse width and bending angles during repeated gripping motion of a robotic hand in nontactile and tactile memory states, respectively. (a) Reproduced with permission.^{[ 127 ]} Copyright 2018, American Association for the Advancement of Science. (b,c) Reproduced with permission.^{[ 128 ]} Copyright 2022, Springer Nature. (d,e) Reproduced with permission.^{[ 129 ]} Copyright 2022, American Association for the Advancement of Science. (f,g) Reproduced with permission.^{[ 68 ]} Copyright 2023, American Association for the Advancement of Science.

Figure 8

Application of NAS in neural interface systems a) Hybrid reflex arc made by integrating an artificial afferent nerve with a biological motor nerve (leg of a discoid cockroach). b) Schematic illustration showing the structure of the artificial sensorimotor system connected with a mouse sciatic nerve. c) Schematics of an artificial sensorimotor loop and photographs showing leg twitching angles in a mouse in response to different frequency stimulations correlating to different pressure inputs. d) Illustration of the structure and stretchable components of SNEN for proprioceptive feedback. e) Images of a mouse with SNEN attached to the leg to enable electrical stimulation of its extensor and flexors. f) Stimulation of the extensor and flexor of the leg with two artificial efferent nerves. (a) Reproduced with permission.^{[ 41 ]} Copyright 2018, American Association for the Advancement of Science. (b, c) Reproduced with permission.^{[ 142 ]} Copyright 2023, American Association for the Advancement of Science. (d‐f) Reproduced with permission.^{[ 143 ]} Copyright 2023, Springer Nature.