All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition

Abstract

With the advent of the Internet of Things era, the detection and recognition of moving objects is becoming increasingly important¹. The current motion detection and recognition (MDR) technology based on the complementary metal oxide semiconductor (CMOS) image sensors (CIS) platform contains redundant sensing, transmission conversion, processing and memory modules, rendering the existing systems bulky and inefficient in comparison to the human retina. Until now, non-memory capable vision sensors have only been used for static targets, rather than MDR. Here, we present a retina-inspired two-dimensional (2D) heterostructure based retinomorphic hardware device with all-in-one perception, memory and computing capabilities for the detection and recognition of moving trolleys. The proposed 2D retinomorphic device senses an optical stimulus to generate progressively tuneable positive/negative photoresponses and memorizes it, combined with interframe differencing computations, to achieve 100% separation detection of moving trichromatic trolleys without ghosting. The detected motion images are fed into a conductance mapped neural network to achieve fast trolley recognition in as few as four training epochs at 10% noise level, outperforming previous results from similar customized datasets. The prototype demonstration of a 2D retinomorphic device with integrated perceptual memory and computation provides the possibility of building compact, efficient MDR hardware.