Embodied neuromorphic intelligence

Abstract

The design of robots that interact autonomously with the environment and exhibit complex behaviours is an open challenge that can benefit from understanding what makes living beings fit to act in the world. Neuromorphic engineering studies neural computational principles to develop technologies that can provide a computing substrate for building compact and low-power processing systems. We discuss why endowing robots with neuromorphic technologies - from perception to motor control - represents a promising approach for the creation of robots which can seamlessly integrate in society. We present initial attempts in this direction, highlight open challenges, and propose actions required to overcome current limitations.

Fig. 1. Robots with end-to-end neuromorphic intelligence.

Some non exhaustive examples of perception (magenta), intelligent behaviour (green) up to action execution (blue) that would all be implemented by means of dedicated Spiking Neural Network (SNN) hardware technology. iCub picture ^©IIT author Agnese Abrusci.

Fig. 2. Neuromorphic sensing for robots.

a the iCub robot (picture ^©IIT author Duilio Farina) is a platform for integrating neuromorphic sensors. Magenta boxes show neuromorphic sensors that acquire continuous physical signals and encode them in spike trains (vision, audition, touch). All other sensors, that monitor the state of the robot and of its collaborators, rely on clocked acquisition (green boxes), that can be converted to spike encoding by means of Field Programmable Gate Arrays (FPGAs) or sub-threshold mixed-mode devices. b The output of event-driven sensors can be sent to Spiking Neural Networks (SNNs) (with learning and recurrent connections) for processing. VISION box in (a): Event-driven vision sensors produce “streams of events” (green for light to dark changes, magenta for dark to light changes). The trajectory of a bouncing ball can be observed continuously over space, with microsecond temporal resolution (black rectangles represent sampling of a 30 fps camera). Table: Event-driven vision sensors evolved from the Dynamic Vision Sensor (DVS) with only “change detecting” pixels - to higher resolution versions with absolute light intensity measurements. The Dynamic and Active pixel VIsion Sensor (DAVIS) acquires intensity frames at low frame rate simultaneously to the “change detection” (with minor cross talk and artefacts on the event stream during the frame trigger). The Asynchronous Temporal Imaging Sensor (ATIS) samples absolute light intensity only for those pixels that detect a change. The CeleX5 offers either frame-based or event-driven readout (with a few milliseconds delay between the two, resulting in loss of event stream data during a frame acquisition). Similar to the DAVIS, the Rino3 captures events and intensity frames simultaneously, however, it employs a synchronised readout architecture as opposed to the asynchronous readout typically found in other event-driven sensors. The ultimate solution combining frames and events is yet to be found. Merging two stand-alone sensors in a single optical setup poses severe challenges in terms of the development of optics that trade-off luminosity with bulkiness. Merging two types of acquisition on the same sensor limits the fill-in factor and increases noise and interference between frames and events.

Fig. 3. AER: example of communication between an event-driven sensor (triangular skin patches, each with 6 sensing areas) and a spiking neural network (SNN) chip.

Each sensing element emits asynchronous spikes that are sent to a bus through arbitration. The same are de-multiplexed to be sent to the correct synapse of the SNN chip.