Single-sample image-fusion upsampling of fluorescence lifetime images

Abstract

Fluorescence lifetime imaging microscopy (FLIM) provides detailed information about molecular interactions and biological processes. A major bottleneck for FLIM is image resolution at high acquisition speeds due to the engineering and signal-processing limitations of time-resolved imaging technology. Here, we present single-sample image-fusion upsampling, a data-fusion approach to computational FLIM super-resolution that combines measurements from a low-resolution time-resolved detector (that measures photon arrival time) and a high-resolution camera (that measures intensity only). To solve this otherwise ill-posed inverse retrieval problem, we introduce statistically informed priors that encode local and global correlations between the two "single-sample" measurements. This bypasses the risk of out-of-distribution hallucination as in traditional data-driven approaches and delivers enhanced images compared, for example, to standard bilinear interpolation. The general approach laid out by single-sample image-fusion upsampling can be applied to other image super-resolution problems where two different datasets are available.

Fig. 1.. Schematic of the LP method.

(A) Shown is a CMOS (fluoresence intensity) FOV, with the SPAD FOV (fluorescence lifetime), overlayed on top of it so as to match the sparse, low–fill factor pixel layout of the SPAD array. a.u., arbitrary units. (B) We zoom in on a 5 × 5 window. All SPAD pixels have a corresponding CMOS measurement, but so do the areas in-between SPAD pixels. We aim to find the lifetime at points with no SPAD samples. For this, we fit a function, for instance, linear interpolation, a cubic spline or a radial basis function Gaussian process. Then, the high-resolution CMOS pixels x_HR that we wish to upsample are fitted with this function, producing a lifetime estimate ${\widehat{\mathrm{\tau }}}_{\text{HR}}.$ (C) We slide the window across the FOV, fitting new functions for each new window and predicting the centers, upsampling the FLIM image to the resolution of the intensity image, window by window.

Fig. 2.. Schematic of the GP method.

(A) Fluorescence intensity of a Convallaria rhizome sample stained with acridine orange, with 8 × 8 sparse lifetime samples overlayed. We extract intensity patches from this image; a few of them correspond to a central lifetime sample. These patches are training data, which we can use to predict the central lifetime of the rest of the patches. (B) Training inputs (patches) are augmented via rotation and mirroring. They can be further augmented by adding the patches that are nearest neighbors of training patches and allocating them the same label (lifetime) as the sampled patch. The deep neural network (DNN) architecture is simple, consisting of three two-dimensional convolutional layers, followed by three fully connected layers. (C) Last, the trained DNN evaluates patches with unsampled centers, thus super-resolving the lifetime image.

Fig. 3.. Upsampling (16 × 16) of MDCK cells.

(A) Low-resolution fluorescence lifetime image (32 × 32) of MDCK cells expressing Flipper-TR dye. (B) Corresponding high-resolution intensity image (512 × 512) of the sample. (C) Windows (5 × 5) of low-resolution FLIM are fitted to corresponding intensity values to generate an LP image (two example windows are shown). (D) A GP image is generated from 13 × 13 intensity patches with central FLIM measurements (two examples are shown). (E) The GT high-resolution FLIM target, intensity-weighted for visualization. (F) The proposed method, upsampling the low-resolution measurement by a factor of 16 × 16. (G) Bilinear interpolation upsampling the FLIM measurement by 16 × 16.

Fig. 4.. Upsampling (8 × 8) of Convallaria images.

(A) Low-resolution fluorescence lifetime image (24 × 32) of a Convallaria rhizome sample stained with acridine orange, viewed under a wide-field microscope. (B) High-resolution intensity image (192 × 256). (C) Example 5 × 5 windows of low-resolution intensity versus FLIM, used for generating the LP shown on the right. (D) High-resolution intensity patches are labeled with lifetime, letting us create a GP. (E to G) GT, 8 × 8 SiSIFUS, and 8 × 8 bilinear interpolation of the data, weighted by local contrast enhanced intensity for visualization (see the Supplementary Materials for details).

Fig. 5.. Upsampling (16 × 16) of SKOV3 cells.

(A) Low-resolution fluorescence lifetime image (16 × 16) of an SKOV3 samples expressing Rac1-Raichu. (B) High-resolution intensity image (256 × 256). (C and D) Examples of local and global dependencies. (E to G) GT, 16 × 16 super-resolved, and 16 × 16 interpolated images.

Fig. 6.. SR (8 × 8) of SKOV3 images.

(A) Low-resolution FLIM image of an SKOV3 cell expressing the Rac1-Raichu probe (24 × 16). (B) The corresponding high-resolution fluorescence intensity image (192 × 128). (C and D) Comparison of local intensity– and global intensity–lifetime dependencies observed in this sample, and the corresponding local and GP images. (E to G) High-resolution GT FLIM, 8 × 8 SiSIFUS, and 8 × 8 bilinearly interpolated images, respectively.