Unsupervised Deep Learning Methods for Biological Image Reconstruction and Enhancement: An overview from a signal processing perspective

Abstract

Recently, deep learning approaches have become the main research frontier for biological image reconstruction and enhancement problems thanks to their high performance, along with their ultra-fast inference times. However, due to the difficulty of obtaining matched reference data for supervised learning, there has been increasing interest in unsupervised learning approaches that do not need paired reference data. In particular, self-supervised learning and generative models have been successfully used for various biological imaging applications. In this paper, we overview these approaches from a coherent perspective in the context of classical inverse problems, and discuss their applications to biological imaging, including electron, fluorescence and deconvolution microscopy, optical diffraction tomography and functional neuroimaging.

Keywords: Deep learning; biological imaging; image reconstruction; unsupervised learning.

Fig. 1.

Overview of self-supervised learning for denoising. Black pixels denote masked-out locations in the images, while 1_J is the indicator function on the indices specified by the index set J.

Fig. 2.

Overview of the self-supervised learning methods for image reconstruction using hold-out masking. Black pixels denote masked-out locations in the measurements and DC denotes the data consistency units of the unrolled network.

Fig. 3.

Denoising results from fluorescence microscopy datasets Fluo-N2DH-GOWT1 and Fluo-C2DL-MSC using a traditional denoising method BM3D and a self-supervised learning method Noise2Self (N2S). We note that supervised deep learning is not applicable as these datasets contain only single noisy images.

Fig. 4.

Reconstruction results from an fMRI application [6] using conventional split-slice GRAPPA technique and self-supervised multi-mask SSDU method [14]. (a) Split-slice GRAPPA exhibits residual artifacts in mid-brain (yellow arrows). Multi-mask SSDU alleviates these, along with visible noise reduction. (b) Temporal SNR (tSNR) maps show substantial gain with the self-supervised deep learning approach, particularly for subcortical areas and cortex further from the receiver coils. (c) Phase maps for the two reconstructions show strong agreement, with multi-mask SSDU containing more voxels above the coherence threshold.

Fig. 5.

Geometric view of deep generative models. Fixed distribution ζ in $\mathcal{Z}$ is pushed to μ_θ in $\mathcal{X}$ by the network G_θ, so that the mapped distribution μ_θ approaches the real distribution μ. In VAE, G_θ works as a decoder to generate samples, while F_ϕ acts as an encoder, additionally constraining ζ_ϕ to be as close to ζ. With such geometric view, auto-encoding generative models (e.g. VAE), and GAN-based generative models can be seen as variants of this single illustration.

Fig. 6.

VAE architecture. F_ϕ encodes x, and combined with random sample u to produce latent vector z. G_θ decodes the latent z to acquire $\widehat{\mathit{x}}$. u is sampled from standard normal distribution for the reparameterization trick. (a) VAE. (b) spatial-VAE [19], disentangling translation/rotation features from different semantics. (c) DIVNOISING [20], enabling superviesd/unsupervised training of denoising generative model by leveraging the noise model p_NM(y|x).

Fig. 7.

Illustration of GAN-based methods for biological image reconstruction. (a) GAN, (b) pix2pix [21], (c) AmbientGAN [22], (d) cryoGAN [23]. x, y denote data in the image domain, and the measurement domain, respectively. G, D refers to generator, discriminator, respectively. H defines the function family of the forward measurement process, parameterized with φ. Networks and variables that are marked in blue have learnable parameters optimized with gradient descent.

Fig. 8.

Geometric view of cycleGAN. $(\mathcal{Y},\nu )$ is mapped to $(\mathcal{X},\mu )$ with G_θ, while H_φ does the opposite. The two mappers, i.e. generators are optimized by simultaneously minimizing d(μ, μ_θ), d(ν, ν_φ).

Fig. 9.

Network architecture of cycleGAN. $G_{\theta }:\mathcal{Y}\mapsto \mathcal{X},H_{\phi }:\mathcal{X}\mapsto \mathcal{Y}$ are the generators responsible for inter-domain mapping. D_X, D_Y are discriminators, constructing ${\mathcal{L}}_{GAN}$. GAN loss is simultaneously optimized together with ${\mathcal{L}}_{c\mathit{y}cle}$

Fig. 10.

ProjectionGAN for the reconstruction of ODT [31]. (a) Conventional Rytov reconstruction via Fourier binning, (b) Gerchberg-Papoulis (GP) algorithm, (c) model-based iterative method using the total variation (TV), and (b) reconstruction via projectionGAN. Artifacts including elongation along the optical axes can be seen in the x − z, y − z cutview of (a),(c). The result shown in (b) is contaminated with resdual noise in the x − z, y − z planes. Result shown in (d) has high-resolution reconstruction without such artifacts, along with boosted RI values.