Minimal approach to neuro-inspired information processing

Abstract

To learn and mimic how the brain processes information has been a major research challenge for decades. Despite the efforts, little is known on how we encode, maintain and retrieve information. One of the hypothesis assumes that transient states are generated in our intricate network of neurons when the brain is stimulated by a sensory input. Based on this idea, powerful computational schemes have been developed. These schemes, known as machine-learning techniques, include artificial neural networks, support vector machine and reservoir computing, among others. In this paper, we concentrate on the reservoir computing (RC) technique using delay-coupled systems. Unlike traditional RC, where the information is processed in large recurrent networks of interconnected artificial neurons, we choose a minimal design, implemented via a simple nonlinear dynamical system subject to a self-feedback loop with delay. This design is not intended to represent an actual brain circuit, but aims at finding the minimum ingredients that allow developing an efficient information processor. This simple scheme not only allows us to address fundamental questions but also permits simple hardware implementations. By reducing the neuro-inspired reservoir computing approach to its bare essentials, we find that nonlinear transient responses of the simple dynamical system enable the processing of information with excellent performance and at unprecedented speed. We specifically explore different hardware implementations and, by that, we learn about the role of nonlinearity, noise, system responses, connectivity structure, and the quality of projection onto the required high-dimensional state space. Besides the relevance for the understanding of basic mechanisms, this scheme opens direct technological opportunities that could not be addressed with previous approaches.

Keywords: delay; dynamical systems; hardware; information processing; machine-learning; pattern recognition; photonics; reservoir computing.

Figure 1

Schematic arrangement of reservoir computing based on a single nonlinear node with delay and time-multiplexing. Virtual nodes are defined as temporal positions along the delay line. Figure has been adapted from Appeltant et al. (2011).

Figure 2

Illustration of the masking steps required for information processing in a delay-based reservoir computer. Figure has been adapted from Appeltant et al. (2011).

Figure 3

Illustration of the input encoding. (A) Temporal sequence of the input signal. (B) Matrix representation of the input signal multiplied by the mask, where the virtual nodes act as a pseudo-space. (C) Temporal sequence of the first sample of the input signal multiplied by the mask, i.e., expanded over the corresponding location of the virtual nodes.

Figure 4

Illustration of the nonlinear transient dynamics. (A) Temporal sequence of the response to the signal depicted in Figure 3C. (B) Matrix representation of the response signal to the input matrix depicted in Figure 3B. Parameter values in Equation (3) are τ = 10, β = 0.7, and ϕ = -π/4, respectively.

Figure 5

Results for a classification task (blue asterisks) and a prediction task (red circles). (A) Performance of the system as a function of the slope of the node-response curve at the operating point. (B) Performance of the system as a function of the curvature of the node response at the operating point. The prediction task (SantaFe time series prediction) was evaluated via the normalized mean square error. The classification task (spoken digit recognition) was evaluated via the misclassification ratio. The results were obtained with the opto-electronic system introduced in Larger et al. (2012).

Figure 6

Classification error rate for a spoken digit recognition task as a function of the reservoir size. The blue line with dots corresponds to a reservoir of N nodes, while the green line with squares corresponds to a reservoir of 400 nodes in which only N nodes are available. The error bars are computed as the standard error over 10 realizations.