Fluorescence Lifetime Endoscopy with a Nanosecond Time-Gated CAPS Camera with IRF-Free Deep Learning Method

Abstract

Fluorescence imaging has been widely used in fields like (pre)clinical imaging and other domains. With advancements in imaging technology and new fluorescent labels, fluorescence lifetime imaging is gradually gaining recognition. Our research department is developing the tauCAM^TM, based on the Current-Assisted Photonic Sampler, to achieve real-time fluorescence lifetime imaging in the NIR (700-900 nm) region. Incorporating fluorescence lifetime into endoscopy could further improve the differentiation of malignant and benign cells based on their distinct lifetimes. In this work, the capabilities of an endoscopic lifetime imaging system are demonstrated using a rigid endoscope involving various phantoms and an IRF-free deep learning-based method with only 6-time points. The results show that this application's fluorescence lifetime image has better lifetime uniformity and precision with 6-time points than the conventional methods.

Keywords: CAPS; convolutional neural networks; endoscopy; fluorescence imaging; fluorescence lifetime imaging; gated camera.

Figure 1

Schematic diagram of the FLT endoscopy imaging.

Figure 2

(a) Intensity pattern of the endoscope illumination through the FoV at WD of 3 cm, and (b) the 1D pattern through the mentioned red line in the diagonal direction, which has a Gaussian distribution. (c) The orange square marks the location of strong illumination, with its corresponding IRF showing a clear signal. (d) The green square indicates a region where the endoscope attenuates the light intensity, corresponding to a noise-dominated IRF.

Figure 3

Resolution analysis of the FLT endoscopy system based on resolution test target USAF 1951.

Figure 4

Topology of FLTCNN to analyze mono-exponential fluorescence decays. The details of hyperparameters in each layer in parenthesis represent the number of filters, and the kernel size, respectively. The input is an image stack of (128,128,6). The architecture of SimiResBlock, and DownSampleBlock (consists of 4, 2D convolutional layers with decrementing filter sizes) are shown with a dashed box. The BN and the ReLU are added after convolutional layers.

Figure 5

Synthetic training data generation flow for mono-exponential fluorescence signal model.

Figure 6

(a) MAE graph of training/validation vs. epochs. (b) The MAE of predicted results of the testing datasets. (c) The mean value of MAE for a lifetime is under different conditions. The SNR takes the value between 20 to 1000 for A = 10, 20, 50, and 100. The blue area denotes the lifetime range of training data. (d) t-SNE visualization was obtained via the last activation map before the down-sampling block, where each point represented a TPSF voxel and was assigned a randomized lifetime value.

Figure 7

FLT image of a uniform ICG-equivalent phantom predicted by (a) Lmfit (Levenberg–Marquardt), (b) FLTCNN in the macroscopic wide-field regime. (c) Shows the normalized fluorescence intensity of the phantom. (d) FLT histogram of the ICG uniform phantom processed with Lmfit and FLTCNN.

Figure 8

(a,b) FLT and intensity images, and (c) FLT histogram of the ICG uniform phantom captured by FLT endoscopy system and processed with FLTCNN algorithm.

Figure 9

(a,b) FLT and intensity images of the uniform ICG phantom under an angle.

Figure 10

(a) Lmfit analysis in the macroscopic regime, (b) Lmfit analysis of 6-time points in the endoscopy regime, (c) FLTCNN analysis in the endoscopy regime, and (d) fluorescence intensity image of the concentration ICG phantom, Quel Imaging. (e–g) histogram of each well related to each approach to predict lifetime.

Figure 11

FLT images of ICG-equivalent phantoms (distortion, coin, and vessel, Quel Imaging) analyzed with (a–c) Lmfit full decay, (d–f) Lmfit 6-data point, (g–i) FLTCNN, and (j–l) fluorescence intensity.

Figure 12

The lifetime distribution of the ICG distortion, coin, and vessel phantoms calculated by (a) Lmfit analysis in the macroscopic regime, (b) Lmfit analysis of 6-time points in the endoscopy regime, and (c) FLTCNN analysis in the endoscopy regime.

Figure 13

(a) Lmfit analysis in the macroscopic regime, (b) Lmfit analysis of 6-time points in the endoscopy regime, (c) FLTCNN analysis in the endoscopy regime, and (d) fluorescence intensity image of QUEL mixed phantoms containing ICG in ST01/LU02 and OTL38 in ST01/LU02. (e–g) histogram of each well related to each approach to predict lifetime.