Bio-inspired cross-modal super-additive plasticity for seamless visual processing-in-sensory and -in-memory

Abstract

Bio-inspired cross-modal visual perception hardware offers potential for edge intelligence. However, physical implementation of such hardware by conventional optoelectronics typically results in linear function combinations, lacking super-additive integration. Here, inspired by the primary cortex of the biological brain, we design a hardware platform based on molybdenum disulfide channel for processing-in-sensory and -in-memory. Cross-modal correlation photoelectric signals processing is demonstrated by utilizing electric field-assisted photogenerated carrier tunneling based on a floating gate photoelectric device array. The devices exhibit high synergistic paradigm super-additive behavior up to 10³ times and significant time-dependent plasticity for visual encoding and perception enhancement. After sensory preprocessing, patterns are accurately routed and recognized by a non-volatile four-transistor ternary content-addressable memory circuit array. The cell maintains a large resistance ratio of 10⁵ and high lookup durability of 10¹². The hardware platform of cross-modal visual perception empowers seamless visual process-in-sensory and -in-memory, providing potential for ubiquitous visual edge intelligent systems.

Fig. 1. Cross-modal correlation perception and design.

a McGurk effect schematic. The auditory perception is influenced by the visual association. The McGurk effect affects auditory perception through vision, producing an effect similar to decoding information. b Cross-modal associative perception of concepts and features: Multi-channel information is coordinated and coupled in the primary cortex, integrating into new perceptions that include the super-additive effect and time dependence of information coordination. c The use of solid-state electronic device arrays to simulate cross-modal correlation response features to achieve efficient information integration and coding. d Simulating the cross-modal correlation response behavior through the electric-field-assisted photogenerated carrier tunneling process, where the use of single-layer molybdenum sulfide helps to generate efficient photogenerated carriers.

Fig. 2. Device structure and basic characteristics.

a Microscope image of the optical sensor array and the 4T-TCAM circuits. Scale bar: 200 μm. The enlarged image of (b) MoS₂ floating-gate transistor cells and (c) 4T-TCAM cell in (a). Scale bar: 50 μm. The channels region magnification of the (d) MoS₂ floating-gate transistor and (e) MoS₂ transistor. Scale bar: 20 μm. f Schematic of the basal component MoS₂ floating-gate transistor. g The cumulative probability distribution of high resistance states (HRS, blue symbol) and low resistance states (LRS, red symbol) for multiple MoS₂ FGTs devices, with current extracted at V_g = 0 V and V_d = 1 V. μ, mean value. σ, standard deviation. h The optical response characteristics under different LED light sources with wavelengths of 470 nm (blue line), 530 nm (cyan line), and 625 nm (red line). The incident light power intensity is 9.5 mW cm⁻². i Schematic of a single TCAM circuit cell based on MoS₂ floating-gate transistors. j Repeated reading of the output current (V_bias = 1 V) of the TCAM cell between searching bit 1 and 0 (the period is 2 s) with storage bit 0 (black line) and 1 (red line). k The read endurance of the 4T-TCAM with storage bit 0 at V_bias = 2 V. Using a 100 ns read pulse width of V_g, applied repeatedly to the MoS₂ transistor. The red symbol represents a matching output, and the black symbol represents a mismatching output. After the endurance test, the resistance ratio can still reach 10⁵ after rewriting storage bit data into the MoS₂ FGTs (blue symbol represents matching output, olive-green symbol represents mismatching output).

Fig. 3. Photoelectric co-modulated responses of MoS₂ FGT sensors.

a Real-time monitored output current by V_g pulse (red line, V_g pulse = −2 V, V_g base = 0 V), LED light (blue line, power intensity is 16.7 mW cm⁻² with λ = 470 nm), and synchronous voltage-LED (orange line) stimulation of 30 times. The inset grayscale images present the output current stimulated by the visual image letter P. b Timing diagrams of applied optical and electrical pulses with the same period and duration. The period is 1 s, and the duration is 10 ms for all stimulations. c Performance comparison of signal-to-noise margin ratio, energy consumption per pixel (data from 20 devices, the error bar represents the standard deviation), and latency for optical-electrical stimulations (1 time) and only optical stimulations (150 times).

Fig. 4. Cross-modal correlation plasticity of MoS₂ FGTs for pattern decoding.

a Real-time monitored output current by synchronous optical-electrical stimulation of 30 times with different ∆t from −10 ms to 10 ms. The period is 1 s and the duration is 10 ms for all optical (power intensity is 16.7 mW cm⁻²) and electrical (V_g pulse = −2 V, V_g base = 0 V) pulses. b–d The energy band diagram presents the tunnel and capture dynamics of photogenerated carriers under different modal stimulation. E_F, Fermi level. E_Fⁿ, electron quasi-Fermi level. E_F^p, hole quasi-Fermi level. e The extracted excitation current with various ∆t for different stimulation times from (a). f Schematic and numerical simulations of an image decoding application based on the integrated sensor array.

Fig. 5. TCAM array simulations for visual security.

a Conventional vision processing architecture based on silicon technologies. ADC, analog to digital converter. CPU, central processing unit. b One-shot processing proposed in this work that monolithically integrates a processing-in-sensor (PIS) scheme combined with a computing-in-memory (CIM) architecture for vision security. c Circuit diagram of the simulated 16 × 4 TCAM array. Each 16 TCAM cell in the 16 × 4 array stores the letters P (R1), K (R2), U (R3), or N (R4), as marked with color blocks. d Returned output resistance from the TCAM array once the digital visual images P, K, U, and N. e Schematic, and simulation results with a blurred image and different keys.