Deep learning-based high-speed, large-field, and high-resolution multiphoton imaging

Abstract

Multiphoton microscopy is a formidable tool for the pathological analysis of tumors. The physical limitations of imaging systems and the low efficiencies inherent in nonlinear processes have prevented the simultaneous achievement of high imaging speed and high resolution. We demonstrate a self-alignment dual-attention-guided residual-in-residual generative adversarial network trained with various multiphoton images. The network enhances image contrast and spatial resolution, suppresses noise, and scanning fringe artifacts, and eliminates the mutual exclusion between field of view, image quality, and imaging speed. The network may be integrated into commercial microscopes for large-scale, high-resolution, and low photobleaching studies of tumor environments.

Fig. 1.

Multiphoton microscopy. a Commercial multiphoton microscope with a fast galvo-resonant scanning system and slow dual-axis galvo scanning system. b Composition of low-quality input and high-quality target images (SHG and TPEF superimposed channels) in datasets. SL: scan lens; TL: tube lens; DM: dichroic mirror; CL: collect lens; OB: objective.

Fig. 2.

Deep learning network architecture. a Overall network architecture, including IW, SAPCD, and RRDAB modules, convolution layers, the skip connection, up-sampling operation, and discriminator. b Image registration framework. Image warping is performed by the ORB feature extraction method to construct the training dataset. The self-alignment pyramid, cascading, and deformable convolutions (SAPCD) are embedded in the generator. c RRDAB reconstruction network. Cascaded RRDAB modules are used for image reconstruction, and dense connectivity DAB for feature communication. The attention blocks consist of two SE modules. DConv: deformable convolution; L: level; LR: low-resolution; HR: high-resolution; DAB: dense attention block; IW: image warp; FA: feature alignment; HR: high resolution; GT: ground truth.

Fig. 3.

Deep learning-enhanced multiphoton images with different scanners. a-c Input, result, and target GT large images, respectively. ROI₁: superimposed channel; ROI₂: TPEF channel; ROI₃: SHG channel. d Intensity profiles along the solid line in ROI₃. e PSNR of input and output images for n = 10 large images; error bars indicate the mean standard deviation (SD). f Comparison of network inference and GT acquisition time. The values of each scale bar are marked in a.

Fig. 4.

Deep learning enables large-field and high-resolution multiphoton imaging. a-f SHG- and TPEF-superimposed channels of input, result, and target GT large images. g Comparison of network inference and GT acquisition time for n = 15 large images. h Curve fitting of intensity profiles along the solid lines in the ROIs of a-c. i NIQE and PIQE of input and output images for n = 200 image tiles; error bars show the mean standard deviation (SD). GT: ground truth.

Fig. 5.

Deep learning enables low photobleaching imaging. a-c Low-power excitation input and network’s output of SHG (cyan), TPEF (magenta), and superimposed channels. d-f ROIs of different sizes in a-c. g, h Intensity profiles along the solid line in d, e. i, j ROIs of background areas in f and the corresponding FFT result. N: noise; S: signal; SFA: scanning fringe artifacts; FFT: fast Fourier transform.

Fig. 6.

Evaluation of IW module on image reconstruction. a Schematic of image misalignment and correction (cyan channel: SHG; magenta channel: TPEF). b Results of registered training, non-registered training, and comparison of RRDAB and RCAN.