Alternation of inverse problem approach and deep learning for lens-free microscopy image reconstruction

Abstract

A lens-free microscope is a simple imaging device performing in-line holographic measurements. In the absence of focusing optics, a reconstruction algorithm is used to retrieve the sample image by solving the inverse problem. This is usually performed by optimization algorithms relying on gradient computation. However the presence of local minima leads to unsatisfactory convergence when phase wrapping errors occur. This is particularly the case in large optical thickness samples, for example cells in suspension and cells undergoing mitosis. To date, the occurrence of phase wrapping errors in the holographic reconstruction limits the application of lens-free microscopy in live cell imaging. To overcome this issue, we propose a novel approach in which the reconstruction alternates between two approaches, an inverse problem optimization and deep learning. The computation starts with a first reconstruction guess of the cell sample image. The result is then fed into a neural network, which is trained to correct phase wrapping errors. The neural network prediction is next used as the initialization of a second and last reconstruction step, which corrects to a certain extent the neural network prediction errors. We demonstrate the applicability of this approach in solving the phase wrapping problem occurring with cells in suspension at large densities. This is a challenging sample that typically cannot be reconstructed without phase wrapping errors, when using inverse problem optimization alone.

Figure 1

Overview of the data processing performed in the alternation approach.

Figure 2

Results of the alternation reconstruction method obtained on the synthetic validation set at low density (358 cells/${\text{mm}}^2$), medium density (909 cells/${\text{mm}}^2$) and high density (1627 cells/${\text{mm}}^2$). Subfigure (a) presents ground truth images (L and A) of the object, subfigure (b) presents holograms obtained by using Eq. (2). Results of first (old) holographic reconstruction (c), of the CNN step (d), and of the final (new) reconstruction (e). Images are $150\times 150$ pixels crops of the original images ($1.67\mathrm{\mu }\text{m}$ pitch). In (c,d,e), peak signal-to-noise ratio (PSNR) measurements are indicated in the top left of the results to assess the reconstruction versus ground truth.

Figure 3

Reconstructed $\mathrm{\Delta }{n}_{r\_recons}$ and $\mathrm{\Delta }{n}_{i\_recons}$ as a function of the ground truth values. The three lines of the figure correspond to three different cell densities, namely 358, 909 and 1627 cells/${\text{mm}}^2$ (see corresponding images in Fig. 2). The results obtained after each individual step of the alternation approach are shown, namely the first reconstruction (blue dots), the CNN output (orange dots) and the final reconstruction (green dots). The results of the linear regressions are indicated with values of slope and coefficient of determination ($R^2$).

Figure 4

Application of the alternation reconstruction method on PC3 non-adherent cells at low density. Estimated number of cells $\approx 8450$, corresponding to a volumetric density of $\approx 14\times {10}^6$ cells/ml (measured in a $20\mathrm{\mu }\text{m}$ thick chamber) or a surface density of $\approx 290$ cells/${\text{mm}}^2$. (a) Full field of view, final reconstruction. (b) Reconstructions of 6 selected regions of interest from image (a) and comparisons with their fluorescence microscope acquisitions. (c–g) Reconstruction of the seventh region of interest, with corresponding raw acquisition (c), first (old) reconstruction result (d), CNN output result (e), final (new) reconstruction result (f) and the comparison with the fluorescence acquisition (g). (h) OPD profiles through one cell (red line 1 in (b)). The maximum OPD on the final reconstruction is about 1000 nm which corresponds to a phase shift of about 5$\pi$. (i) OPD profiles through two cells (red line 2 in (b)). (j) OPD profile through two cells (red line in (f)). The black arrows indicate CNN errors (orange curve in (j) and red arrows in (e)) that are corrected by the last reconstruction (green curve).

Figure 5

Application of the alternation reconstruction method on PC3 non-adherent cells at high density. Estimated number of cells: $\approx 34,000$ corresponding to a volumetric density of $\approx 57\times {10}^6$ cells/ml (measured in a $20\mathrm{\mu }\text{m}$ thick chamber) or a surface density of $\approx 1140$ cells/${\text{mm}}^2$. (a) Full field of view final reconstruction. (b) Reconstructions of 6 selected regions of interest (a) and their comparisons with the fluorescence microscope acquisitions. (c–g) Reconstruction of the seventh region of interest, with corresponding raw acquisition (c), first (old) reconstruction result (d), CNN result (e), final (new) reconstruction result (f) and the comparison with the fluorescence acquisition (g). Red boxes in (e–g) highlight the discrepancies between the two modalities.

Figure 6

Results of the alternation reconstruction method applied to adherent PC12 cells treated with neuron growth factor. The first reconstruction results is shown in (a), CNN result in (b) and final reconstruction result in (c). (d–o) present detailed results corresponding to three regions of interest depicted by red boxes in (a). (g) OPD profile through two neuron cell bodies (red line in (d)). (k) OPD profile through two dendrites (red line in (h)). (o) OPD profile measured at the proximity of a cell, through two dendrites (red line in (l)).

Figure 7

Principle of lens-free microscopy. A 2D object is illuminated by a partially coherent light. The intensity of the generated interference pattern at a distance Z behind the sample is recorded with a camera.

Figure 8

Overview of the synthetic data generation method. A pair of synthetic images (ground truth) is generated, representing cells in suspension (L denotes the optical path difference and A the absorption). Using Eq. (2), the simulated intensity measurement image I is obtained for a given sample-to-sensor distance Z. The first reconstruction applied to I generates the simulated images $L_{\mathit{recons}}$ and $A_{\mathit{recons}}$. The presented images are $400\times 400$ pixels ($1.67\mathrm{\mu }\text{m}$ pitch) crops of an image with ${10}^3$ cells corresponding to a density of 358 cells/${\text{mm}}^2$.

Figure 9

Design of the 20 layers convolutional neural network used for unwrapping of the holographic reconstruction. Each layer consists of three sub-layers, $(5\times 5)$ convolution layers, a batch normalization layer and a ReLU activation layer. No dimension changes are performed inside the network.