Seeing through arthropod eyes: An AI-assisted, biomimetic approach for high-resolution, multi-task imaging

Abstract

Arthropods have intricate compound eyes and optic neuropils, exhibiting exceptional visual capabilities. Combining the strengths of digital imaging with the features of natural arthropod visual systems offers a promising approach to harness wide-angle vision and depth perception while addressing limitations like low resolving power. Here, we present an artificial intelligence-assisted biomimetic system modeled after arthropod vision. We developed a biomimetic compound eye camera with an effective pixel number of 4.3 megapixels capable of producing full-color panoramic images with a viewing angle of 165° and resolving power of 40 micrometers. Using rich visual information, our system achieves high-fidelity image reconstruction, precise 3D position prediction, high-accuracy classification, and pattern recognition through a multistage neural network. Moreover, our compact biomimetic visual system can simultaneously track the 3D motion of multiple miniature targets independently. The proof-of-concept biomimetic arthropod visual system offers a computational panoramic imaging solution, advancing applications in industry, medicine, and robotics.

Fig. 1.. BCE and artificial visual system.

(A) Scenario of an arthropod surrounded by predators and prey. (B) Illustration of a natural visual system consisting of a compound eye and a brain. (C) Illustration of an artificial visual system consisting of the BCE camera and digital signal processor. (D) Exploded view of the BCE camera. (E) Image of the BCE. CMOS, complementary metal-oxide semiconductor. (F) Scanning electron microscopy image of the BCE. (G) Image of the BCE camera.

Fig. 2.. Optical characterization of the BCE.

(A) The ASF of the central ommatidium (α = 0°, β = 0°). (B) The ASF of the ommatidia with the polar angle of 68.73°. (C) The ASF of the BCE. (D) Measurement of the ASF for the BCE. The ASF was estimated by measuring the response of the ommatidia under the illumination of the collimated light with different incident angles. Each data point stands for the summation of the measured light intensity for the incidence from a specific direction. The surface is generated by interpolating the scattered data points. (E) Experiment setup for measuring the optical properties of the ommatidia. (F) The intensity distribution at the proximal ends of the optical waveguides of the ommatidia with the orientations of (α = 0°, β = 0°) and (α = 68.73°, β = 180°). (G) Evaluation of the optical cross-talk among the ommatidia. (H) Simulation of the light propagation in the ommatidia and the images captured at the proximal ends of the optical waveguides.

Fig. 3.. The multistage multi-task visual processing.

(A) The handwritten digits used for the masks. (B) The five colors of the light source. (C) The 3D coordinate system. (D) The procedure of the deep learning based 3D positioning, image reconstruction, color classification, and pattern recognition.

Fig. 4.. The biomimetic arthropod visual system and binocular imaging system.

(A) The experimental setup of the imaging system. (B) The top view of the imaging systems. (C) The measurement of the spatial resolution of the binocular imaging system and the intensity profile along the green line. The resolution target is placed in front of the system [green point in (B)]. (D and E) The measurement of the spatial resolution of the biomimetic arthropod visual system and the intensity profiles along the red and blue lines. The resolution targets are placed at [α = 0°, β = 0°; red point in (B)] and [α = 69°, β = 180°; blue point in (B)], respectively.

Fig. 5.. Panoramic imaging and artificial visual cognition using the BCE camera.

(A) Imaging and recognition of the blue handwritten digit “8” placed on the side of the BCE camera. (B) Imaging of the green and yellow handwritten digits “6” placed at different distances to the BCE camera. (C to E) Detection and reconstruction of the geometry, alphabet, and insect patterns. GT, ground truth; Pred., prediction.

Fig. 6.. Tracking the two digits using the BCE camera.

(A) Schematic diagram of the tracking experiments. The mask with the orange digit “0” was fixed, while the mask with the yellow digit “8” moved away from the center. (B) The spatial positions of the two digits and the movement path of the digit “8.” The yellow and black solid dots are the target and predicted positions for the digit “8,” respectively. The orange solid dots and the black circles are the target and predicted positions for the digit “0,” respectively. (C to E) The images captured by the BCE camera and the reconstructed digit patterns, (i) digit “8” and (ii) digit “0,” at different moments. (F to H) 3D views of mask movement reconstructed by the visual system. The orientation is truly presented while the scale is arbitrary. The 3D view with true scale is shown in fig. S26.