Nonlocal flat optics for size-selective image processing and denoising

Abstract

All-optical image processing based on metasurfaces is a swiftly advancing field of technology, due to its high speed, large integrability and inherently low energy requirements. So far, the proposed devices have been focusing on canonical operations, such as differentiations to perform edge detection across all objects in a complex scene. Yet, undesired background noise and clutter can hinder such operations, requiring target selection with digital post-processing which inherently limits the overall accuracy, efficiency and speed. Here, we introduce an optical solution for real-time size-selective image processing and experimentally demonstrate the concept with a metal-dielectric-metal film performing a spatial band-pass filter in momentum space. We show high-resolution (~0.9 μm) edge detection and real-time dynamic denoising, ideally suited for bio-imaging applications and target recognitions. Our demonstrated k-space filtering metasurface expands the scope of nonlocal flat optics for analog image processing, ushering in opportunities for ultra-compact, cost-effective, and multifunctional image processors.

Fig. 1. Schematic of a size-selective imaging processor acting as a rectangular band-pass filter operator.

A conceptual illustration of a nonlocal flat optics-based filter, designed to execute the rectangular band-pass OTF (H(k_r)), capable of modulating and filtering specific spatial frequencies associated with targets of different sizes. This modulation is achieved by configuring the band-pass numerical aperture (NA) range (NA₁–NA₂). Such customization facilitates differentiated image processing, including size-selective edge detection (red, yellow, and green parts), specifically, high NA edge detection (green part), and denoising (blue part) tailored to different targets. For example, the red part can detect the edge of the larger image ‘C’ but allows images with other sizes to pass through unchanged; similarly, the yellow part applies to image ‘B’, and the green part applies to the smaller image ‘A’. Meanwhile, the blue part effectively reduces noise without altering the images ‘A’, ‘B’, and ‘C’.

Fig. 2. Size-selective image processor using a nonlocal MDM film.

a The transmission principle involves light incidence at an angle θ_i into the MDM film, consisting of Ag and MgF₂ layers with phase accumulation δ. Simulation of electric fields at various incidence wavevectors at a 650 nm wavelength showcases that the nonlocal effect is observable only when the phase is matched (NA = 0.5), leading to maximum transmission, while conversely, transmission is nearly zero for NA of 0 and 0.9. b Simulated color-coded transmission coefficient t_pp(λ, k_r), showcases a second-order resonance within the visible light for p polarization. The selected representative wavelengths include blue (λ = 466 nm), green (λ = 532 nm), and red (λ = 650 nm) light, indicated by the corresponding-colored dashed lines. c The designed OTFs demonstrate Gaussian-type band-pass filtering at three operating wavelengths with NA ranges of 0–0.8 NA, 0.43–0.6 NA, and 0.93–0.99 NA. The solid-colored lines depict the simulated OTFs, and the dashed lines represent the corresponding ideal rectangular band-pass OTFs. The simulation results demonstrate its size-selective denoising and edge detection capabilities. d Photo of the fabricated two-inch size-selective image processor (yellow dashed box). e Wavevector-dependent transmission spectra (|t(k_r)|²) by ellipsometry at 466 nm, 532 nm, and 650 nm wavelength under p-polarized light (dashed lines), which align well with the simulation results (solid lines).

Fig. 3. Characterizing size-selective imaging processor with artificial targets.

a The experimental resolution of the size-selective imaging processor under red (λ = 650 nm), green (λ = 532 nm), and blue (λ = 466 nm) lights. Under red and green lights, the processor demonstrates selective edge detection with sizes ranging from 1.6 to 2.3 μm and 0.9 to 1.6 μm, respectively. The insets depict the ideal optical transfer function of the image processor (dashed line) and the measured optical transfer function (solid line) at different operating wavelengths. Furthermore, under blue light, the processor efficiently engages in selective denoising of noise with a size less than 0.3 μm. Conversely, it preserves information when the noise size is 0.4 μm, as depicted in the inset that provides a magnified view of the retained noise. b The processor’s size-selective edge detection capability with artificial targets of diverse letters and shapes under red and green lights, encompassing the letters ‘A’ to ‘I’, the optical field vector diagram denoted as ‘E–H–K’, and the ‘Schrödinger staircase’. Specifically, under red light, the processor discernibly enhances the edges of letters ‘D’ to ‘I’ (size: 1.6–2.3 μm) and the letter ‘H’ with its axis (size of 2.2 μm). Conversely, under green light, the edge enhancement is observed on letters ‘A’ to ‘D’ (size: 1–1.6 μm) and the letter ‘E’ with its axis (size = 1.4 μm). Moreover, the ‘Schrödinger staircase’ achieves its artistic effect by designing the sizes of the upper (size of 2.3 μm) and side (size of 1.2 μm) surfaces. Consequently, the staircase imparts downward and upward visual effects under red and green light, respectively. The insets display locally magnified details of the structural features of the edges. c The size-selective denoising capability of the processor through the examination of the letters ‘RGB’, the ‘satellite’, and a QR code, with the size of the noise being 0.3 μm. In comparison to the results obtained through bright field imaging, the noise surrounding the letter ‘B’, the ‘satellite’, and the QR code is efficiently eliminated following the denoising.

Fig. 4. Characterization of real-time size-selective imaging in dynamic living biological cells.

Size-selective edge detection experiments on yeast cells (size ~3 μm, (a)) and Simmental cattle sperm cells (size ~1 μm, (b)), both individually and in their mixed state (c). The edges of yeast cells are significantly enhanced under red light but remain nearly unchanged under green light. Conversely, the edges of sperm cells are significantly enhanced under green light but remain nearly unchanged under red light. The insets provide localized magnification results of selected cellular structures, highlighting effectively enhanced edges with yellow dashed boxes and the nearly unchanged parts with white dashed boxes. d Size-selective denoising experiments involving yeast cells surrounded by silver particles revealed a substantial impact on the imaging results in the absence of a processor. However, upon the insertion of the processor under identical imaging conditions, the noise is markedly reduced. The white and yellow dashed boxes highlight zoomed-in views of the same areas in the image, one with the processor and one without, respectively. e Size-selective denoising experiments on dynamic living nanobacteria distributed both inside and outside yeast cells, with a significant and time-dependent change in the location of the living nanobacteria outside the cells. In the insets, localized enlargements of the same nanobacteria are presented, showcasing images without and with the addition of the processor, demarcated by a white dashed box and a yellow dashed box. These insets on the right offer zoomed-in views of the nanobacteria, displaying positional shifts and tracked images at three different moments, both without and with the processor in place, respectively. This highlights the processor’s capability to remove noise and enhance the clarity of dynamic living nanobacteria imaging in real-time.

Fig. 5. Improvement of target recognition performance by size-selective image processing and denoising.

a The accuracy comparison between uniform edge-detected images (black line) and size-selective edge-detected images (red line) shows that size-selective edge detection achieves 90% recognition accuracy after 59 epochs, while uniform edge detection requires 112 epochs to reach the same level (green line). b Comparison of the recognition accuracy of the noisy images (black line) and the denoised images (red line). The accuracy of the noisy image is about 40%, while the accuracy of the denoised image rapidly increases to 96%. This shows that denoising significantly improves recognition accuracy. c The recognition accuracy of the uniform edge-detected images with noise (blue line), size-selective edge-detected images with noise (black line), and size-selective edge-detected images after denoising (red line) is 90%, 33%, and 22.5%, respectively. Representative datasets for the tasks are shown at the bottom.