Neuromorphic computing for robotic vision: algorithms to hardware advances

Abstract

Neuromorphic computing offers transformative potential for AI in resource-constrained environments by mimicking biological neural efficiency. This perspective article analyzes recent advances and future directions, advocating a system design approach that integrates specialized sensing (e.g., event-based cameras), brain-inspired algorithms (SNNs and SNN-ANN hybrids), and dedicated neuromorphic hardware. Using vision-based drone navigation (VDN) as an exemplar-drawing parallels with biological systems like Drosophila-we demonstrate how these components enable event-driven processing and overcome von Neumann architecture limitations through near-/in-memory computing. Key challenges include large-scale integration, benchmarking standardization, and algorithm-hardware co-design for emerging applications, which we discuss alongside current and future research directions.

Fig. 1. A comprehensive neuromorphic computing pipeline for vision-based drone navigation (VDN), spanning from sensing to learning to hardware implementation.

(Top-left) Inputs and Sensing: Depiction of frame- and event-based inputs for multiple moving objects, demonstrating how traditional frame cameras capture analog intensity information while event cameras detect motion-induced intensity variations. The biological pathway shows the human eye, illustrating how biological systems perceive changes in intensity and color. The neuromorphic sensing section displays both frame-based cameras and event-based cameras working together to capture similar information. (Top-right) Neural Processing: The top biological networks section depicts neurons with synapses and interconnected neural pathways, representing the brain’s highly parallel and recurrent connections that perform computations within memory itself for exceptional efficiency. The central neuromorphic algorithms section shows various neuron types including ReLU and LIF (Leaky Integrate-and-Fire) neurons, alongside different network architectures: ANNs (Artificial Neural Networks), SNNs (Spiking Neural Networks), and Hybrid SNN-ANN configurations that combine the best of both approaches for optimal algorithmic accuracy and efficiency. The hardware section presents conventional processors (CPU, GPU, FPGA) alongside neuromorphic-specific architectures including NVM-based IMC (Non-Volatile Memory-based In-Memory Computing) and hybrid hardware solutions, utilizing device technologies such as RRAM and STT-MRAM that can efficiently implement synaptic memories and work with CPU/GPU architectures for improved efficiency and latency. (bottom) Practical implementation for vision-based drone navigation (VDN), showcasing four key vision tasks: optical flow, depth estimation, segmentation, and object detection.

Fig. 2. Different architectural designs for optical flow estimation using event and/or frame cameras.

a Outputs from frame-based, event-based, and sensor-fused frame+event cameras. b U-Net based Encoder-Residual-Decoder architectures for Fully-ANN, Fully-SNN and Hybrid SNN-ANN topologies, employed for optical flow estimation. The red block depicts the output accumulator for aggregating SNN outputs over time. c Sensor-fused hybrid SNN-ANN architecture using both event and frame modalities for estimating optical flow. (Adapted with permission from Fusion-FlowNet. Copyright 2022, IEEE). The architecture consists of SNN and ANN encoder blocks to handle event and frame inputs respectively.

Fig. 3. Performance comparison between state-of-the-art architectures for optical flow estimation on the MVSEC dataset.

SOTA baselines (EvFlow-Net, Spike-FlowNet and Fusion-FlowNet have their average endpoint error (AEE) depicted on the left. The right side compares the AEE between fully-ANN and fully-SNN architectures with decreasing model sizes. (Data taken with permission from Adaptive-SpikeNet. Copyright 2023, IEEE).

Fig. 4. Architecture for lightweight SNN-based object detection pipeline - DOTIE.

A single-layer SNN utilizing its inherent leak to filter out events moving at a speed greater than a specified value. These speed-separated events are then clustered to obtain a bounding box for object detection. (Adapted with permission from DOTIE. Copyright 2023, IEEE).

Fig. 5. Representative neuromorphic hardware architectures, highlighting key differences in dataflow, memory integration, and compute organization across von Neumann and non-von Neumann systems.

(Top) Simplified architecture of a CPU, a GPU, and a custom multi-chip accelerator. CPUs and GPUs follow von Neumann architecture, where computation is separate from memory. On the other hand, a large number of custom hardware accelerators follow a 2-D mesh-like hierarchical design with memory and computation units distributed across the whole fabric. (Top right) The design of a SpiNNaker node is shown where each node uses a packet router to communicate spikes using a routing table. Multiple such nodes can be connected to make a massively parallel computing system. (Bottom left) Truenorth uses a 2-D mesh network of neurosynaptic cores that have crossbar-like connectivity where input spikes are mapped at the rows, each column represents an output neuron, and the crosspoint represents the synaptic connections. (Bottom right) A generalized representation of a hierarchical in-memory computing core. Each core in the figure consists of multiple compute blocks, and each compute block contains multiple in-memory computing arrays arranged in a crossbar format. An analog in-memory computing array is depicted in the figure with digital-to-analog converters (DACs) at the row periphery and a shared analog-to-digital Converter (ADC) at the column periphery.