Nonlinear optical encoding enabled by recurrent linear scattering

Abstract

Optical information processing and computing can potentially offer enhanced performance, scalability and energy efficiency. However, achieving nonlinearity-a critical component of computation-remains challenging in the optical domain. Here we introduce a design that leverages a multiple-scattering cavity to passively induce optical nonlinear random mapping with a continuous-wave laser at a low power. Each scattering event effectively mixes information from different areas of a spatial light modulator, resulting in a highly nonlinear mapping between the input data and output pattern. We demonstrate that our design retains vital information even when the readout dimensionality is reduced, thereby enabling optical data compression. This capability allows our optical platforms to offer efficient optical information processing solutions across applications. We demonstrate our design's efficacy across tasks, including classification, image reconstruction, keypoint detection and object detection, all of which are achieved through optical data compression combined with a digital decoder. In particular, high performance at extreme compression ratios is observed in real-time pedestrian detection. Our findings open pathways for novel algorithms and unconventional architectural designs for optical computing.

Keywords: Applied optics; Imaging and sensing; Optical techniques; Photonic devices; Techniques and instrumentation.

Fig. 1. Concept of using a multiple-scattering cavity as a passive, tunable nonlinear optical information processor.

a, Experimental setup in which the key component for creating the passive nonlinear random mapping is a DMD mounted on an integrating sphere. The output of the cavity produces a fully developed speckle pattern, with its response being nonlinear in the geometric configuration of the DMD. b, Representative figure showing that the cavity essentially encodes the input pattern on the DMD by optically mixing different areas of input through multiple bounces to create a highly nonlinear feature—a speckle recorded by a camera (input pattern is adapted from the MNIST dataset). c, Mathematical representation of a nonlinear mapping process that transforms a set of input elements on the DMD into a collection of nonlinear features in the output speckle pattern. Multiple scatterings in the cavity generate mixed terms of input values at different pixels with various high nonlinear orders, which provide rich nonlinear features that can be optimally trained to enhance performance in complex computational tasks. f(x) denotes the operation of scaling the configuration of a DMD macropixel x_i,j.

Fig. 2. Classification with nonlinear mapping.

a, Training data from the Fashion MNIST datasets are used to train a one-layer neural network as a digital decoder for classification tasks. Additionally, the percentage of the modulated area on the DMD is changed among 6.25%, 25% and 100% to adjust the order of nonlinear mapping. With full (100%) modulation of DMD, the nonlinear order is further enhanced by covering the output port with a partial reflector (silicon wafer). b, Fashion MNIST classification results with a linear classifier are presented under different numbers of output modes (speckle grains) and varying nonlinear strengths. The optical linear features with quadratic detection are simulated by scattering from a single layer with intensity detection to create a quadratic nonlinear response. Note that a linear regression for binarized Fashion MNIST data cannot exceed 77.6% with the same number of modes. c,d, Violin plots representing the distributions of mutual information between the speckle grains and classification targets under varying numbers of output modes (c) and differing orders of nonlinear mapping by changing the modulated area on the DMD or partially closing the cavity (enhanced) (d). For n speckle mode (n on the x axis), 4n replicated measurements from the same input were performed in c and d. The dashed line plots depict the median values of the mutual information. Each violin’s width reflects the distribution of the mutual information values of the speckle grains and its probability density. Within each violin, the slim black vertical line represents the range of minimum and maximum values; the black box represents the first to third percentile; the white dot represents the median. c, Mutual information analysis when the number of output modes (speckle grains) varies under the highest-order nonlinear mapping. d, Mutual information analysis with low-dimensional speckle features (four output modes) for Fashion MNIST as a function of the nonlinear orders varied by modulated area on the DMD, showing the advantage of going to higher-order nonlinear mapping.

Fig. 3. Computing performance enhanced by nonlinear optical data compression.

a, Concept of image reconstruction using linear optical complex media for linear encoding and camera detection with quadratic response. b, Reconstruction using the speckle features from a. The orange boxes represent the wrongly reconstructed pairs. c, Multiple-scattering cavity as a nonlinear optical encoder along with camera detection and employing compressed speckle features for digital reconstruction of the original image data. d, Reconstruction from speckle features generated by the multiple-scattering cavity. In b and d, approximately 25 speckle grains are used with a compression ratio of 31:1 and are used to train two digital decoders (Methods). It is demonstrated that given the same number of compressed output modes (speckle grains), nonlinear features generated from the cavity can provide a reduced mean squared error by 0.6, resulting in a better reconstruction of the images in d compared with b. More results are provided in Supplementary Figs. 4–6. e, Concept of keypoint detection in human faces (images with 96 × 96 pixels) with compressed speckle features. f, Keypoint detection with a mode compression ratio of 576:1, using 16 output modes with relatively weaker nonlinearity (25% modulated areas in the DMD) and a five-layer MLP decoder. g, Improved keypoint detection with a reduced mean error in pixels across 15 keypoints (1.06 pixels compared with 1.86 pixels errors in f), using 16 output modes (speckle grains) with relatively stronger nonlinearity (full modulated areas in the DMD) and a nine-layer MLP decoder.

Fig. 4. Real-time video pedestrian detection in driving with high mode compression ratio using only 25 output modes.

a, Schematic of real-time pedestrian detection using video data from a dash camera during driving. The multiple-scattering cavity functions as an optical data compressor, and compressed nonlinear optical features are utilized for pedestrian detection with a digital decoder. b, Demonstration of pedestrian detection at a rate close to a real-time video. The magenta boxes represent the inference results from the speckle. The green boxes represent the ground truth. The speed of optical processing, that is, nonlinear feature generation, is as fast as light, and its readout speed is limited by only the camera. With only 25 modes, our camera can currently reach at least 800 Hz. The inference time with the 25 modes in pedestrian detection is 0.0035 s, leading to a total response time (inference + generation of optical features) of less than 0.1000 s, which is faster than the typical human response time of ~0.2000–22.0000 s. The error unit is in pixels (px). c, Demonstration of pedestrian detection at various locations during continuous video streaming; the mean detection error with only 25 modes remains within 1.92 pixels (px).