Label2label: training a neural network to selectively restore cellular structures in fluorescence microscopy

Abstract

Immunofluorescence microscopy is routinely used to visualise the spatial distribution of proteins that dictates their cellular function. However, unspecific antibody binding often results in high cytosolic background signals, decreasing the image contrast of a target structure. Recently, convolutional neural networks (CNNs) were successfully employed for image restoration in immunofluorescence microscopy, but current methods cannot correct for those background signals. We report a new method that trains a CNN to reduce unspecific signals in immunofluorescence images; we name this method label2label (L2L). In L2L, a CNN is trained with image pairs of two non-identical labels that target the same cellular structure. We show that after L2L training a network predicts images with significantly increased contrast of a target structure, which is further improved after implementing a multiscale structural similarity loss function. Here, our results suggest that sample differences in the training data decrease hallucination effects that are observed with other methods. We further assess the performance of a cycle generative adversarial network, and show that a CNN can be trained to separate structures in superposed immunofluorescence images of two targets.

Keywords: Antibody labelling; Cellular structures; Content-aware image restoration; Convolutional neural networks; Fluorescence microscopy; Noise2noise.

Fig. 1.

Qualitative L2L and N2N results for images of actin. (A) Confocal image pair of a fixed HeLa cell that was dual labelled with the anti-β-actin antibody AC-15 and a phalloidin stain, which was excluded from the CNN training. Scale bar: 20 µm. (B) Reconstructed image of AC-15 by a CNN after L2L training with images of AC-15/phalloidin as training input/benchmark, using an L_3S-SSIM loss function. (C) Original and processed images of AC-15 for two ROIs (6 µm×6 µm). From left to right: raw image, restored images after N2N and L2L training with an L₁ and L_3S-SSIM loss function, respectively, and a 20-frame average. (D) The corresponding image of phalloidin and the RMS map between the raw and the predicted image of AC-15 by the network after L2L training.

Fig. 2.

Loss function-dependent L2L and N2N results for images of the microtubule network and caveolae. (A-D) Confocal images of MeT5A cells that were dual labelled with the anti-tubulin antibodies DM1A (A) and YOL1/34 (B), and STED images of MeT5A cells that were dual labelled with the anti-CAVIN-1 antibody D1P6W (C) and the anti-CAV1 antibody 4H312 (D). (A,C) From left to right: raw image of a representative training input, reconstructed images after N2N and L2L training with an L₁ or L_3S-SSIM loss function, and a corresponding 20-frame average or high resolution STED image. (B,D) Representative training benchmarks for L2L training are displayed. Images shown were excluded from the network training. Scale bars: 1 µm (A,B); 200 nm (C,D).

Fig. 3.

Network architecture-dependent L2L results for images of PXN. Confocal images of a HeLa cell dual labelled with the two anti-PXN antibodies 5H11 and Y113, which were used as training input and benchmark for L2L training. (A) From left to right: raw image of 5H11, the restored images by a CNN/CycleGAN after L2L training with paired/unpaired images, and the corresponding image of Y113 (B). Scale bar: 20 µm. (C) Training results for two ROIs (6 µm×6 µm). From left to right: input (5H11), the restored images by a CNN after N2N and L2L training with an L₁ loss function, the restored image by a CycleGAN, and a 20-frame average. (D) Corresponding benchmark images (Y113) for L2L training and its predictions by a CNN after L2L training as outlined above. A CNN that was trained with L2L data in-paints focal adhesions (see white arrowheads) and reduces cytosolic protein (see ROI number 2) for both the training input and benchmark. Images shown were excluded from the network training.

Fig. 4.

Qualitative results after training a CNN to separate cellular structures in superposed images of a nuclear stain and an antibody against a plasma membrane protein. (A) Training input and benchmark images of a MeT5A cell that was dual labelled with the nuclear stain SYTOX Green and an anti-CD44 antibody, and corresponding reconstructions after training a CNN with an L_3S-SSIM. Scale bar: 10 µm. The image pairs were obtained via sequential imaging by changing the excitation wavelength. (B) Qualitative result for an ROI (5 µm×5 µm). Prediction success is dependent on the level of superposition of both labels. Structures appear slightly blurry in the restorations compared to the benchmark, but image noise and jitter are reduced. The images shown were excluded from the training.

Fig. 5.

Repeated cross validation for L2L training. The mean relative change (input/benchmark versus restoration/benchmark) of the NRMSE and 5S-SSIM index after L2L training with image pairs of different cellular structures. Boxes represent 25th and 75th percentiles with median, whiskers represent standard deviations. The image pairs for the trainings were generated of cells that were dual labelled for the actin cytoskeleton (N_tot=68), tubulin (N_tot=51), caveolae (N_tot=60) or PXN (N_tot=77), dependent on the number of raw image pairs that were randomly selected from the total dataset for the cross validation. Each data point is the mean value for an eightfold (actin and PXN) or tenfold (tubulin and caveolae) cross validation that was repeated for small image pair numbers.