Deep learning-based fusion of widefield diffuse optical tomography and micro-CT structural priors for accurate 3D reconstructions

Abstract

Widefield illumination and detection strategies leveraging structured light have enabled fast and robust probing of tissue properties over large surface areas and volumes. However, when applied to diffuse optical tomography (DOT) applications, they still require a time-consuming and expert-centric solving of an ill-posed inverse problem. Deep learning (DL) models have been recently proposed to facilitate this challenging step. Herein, we expand on a previously reported deep neural network (DNN) -based architecture (modified AUTOMAP - ModAM) for accurate and fast reconstructions of the absorption coefficient in 3D DOT based on a structured light illumination and detection scheme. Furthermore, we evaluate the improved performances when incorporating a micro-CT structural prior in the DNN-based workflow, named Z-AUTOMAP. This Z-AUTOMAP significantly improves the widefield imaging process's spatial resolution, especially in the transverse direction. The reported DL-based strategies are validated both in silico and in experimental phantom studies using spectral micro-CT priors. Overall, this is the first successful demonstration of micro-CT and DOT fusion using deep learning, greatly enhancing the prospect of rapid data-integration strategies, often demanded in challenging pre-clinical scenarios.

Fig. 1.

(a) Snapshot of some of the EEMNIST characters used for training. (b) An example in silico phantom with a random embedding with a reduced scattering coefficient, ${\mu }_{s^{\prime }}^e$ , of $1$ $m{m}^{-1}$ and an absorption coefficient, ${\mu }_a^e$ , of $0.008$ $m{m}^{-1}$ . The background reducing scattering coefficient, ${\mu }_{s^{\prime }}^b$ , and background absorption coefficient, ${\mu }_a^b$ , are $1$ $m{m}^{-1}$ and $0.004$ $m{m}^{-1}$ , respectively. Hence, the $\delta {\mu }_a$ value is $0.004$ $m{m}^{-1}$ . These values are relevant to soft tissues. The embedding is placed at a depth of $6$ mm from the illumination plane. (c) Illumination and detection bar patterns acquired experimentally and used on MCX to generate the measurement vectors. (d) Widefield transmission setup on MCX, showing the illumination and detection planes. Two cylinders (with different OPs) have been placed as embeddings for visualization. (e) Simulated perturbed ( $\varphi$ , shown in red) and unperturbed ( ${\varphi }_o$ , shown in blue) measurement vectors obtained from MCX for the sample embedding in (b).

Fig. 2.

(a) The Modified AUTOMAP (ModAM) architecture used for 3D reconstructions in widefield DOT. For our work, the network inputs a Rytov-normalized measurement vector and outputs a 3D $\delta {\mu }_a$ reconstruction map of dimension $30\times 40\times 20$ $m{m}^3$ . ReLU activation follows each convolutional layer. (b) Typical training and validation loss curves obtained by training the ModAM network.(c) A set of $10$ validation curves obtained using training the same network $10$ times by shuffling the training dataset.

Fig. 3.

(a) The proposed Z-AUTOMAP architecture. It consists of a bi-directional flow where the convulational layers are concatenated to combine features extracted from the normalized 3D micro-CT mask (for training purposes, they are the GT intensity masks derived from the same EEMNIST character from which the Rytov measurement is calculated) and the Rytov measurement vector. The output of the network is again a 3D $\delta {\mu }_a$ reconstruction map of dimension $30\times 40\times 20$ $m{m}^3$ . ReLU activation is applied after each convolutional layer. (b) Typical training and validation loss curves obtained by training the Z-AUTOMAP network.(c) A set of $10$ validation curves obtained using training the same network $10$ times by shuffling the training dataset.

Fig. 4.

(a) The obtained in silico results for a phantom with a single embedding placed at a depth of $2mm$ from the plane of illumination in terms of the iso-volume, 2D cross-sections at a depth of $2$ mm, and a graph showing the average distribution of $\delta {\mu }_a$ values over the 2D cross-sections. The results are tabulated quantitatively in terms of the MSE, the VE, and the MS-SSIM. (b) The results for a phantom with 3 embeddings placed at the same depth of $2$ mm and having the same $\delta {\mu }_a$ value. The results for LSQ are produced at an iso-volume $30$ % of the maximum value. (c) Reconstructions results for 3 embeddings placed at 3 different depths from the plane of illumination. Each embedding has a different $\delta {\mu }_a$ value. The black lines represent the GT $\delta {\mu }_a$ values for each embedding. All the results for are produced at an iso-volume of $30\mathrm{\%}$ of the maximum value.

Fig. 5.

A simplified schematic of the experimental setup. It consists of an illumination DMD and a detection DMD in transmission configuration (along with their associated optics). The illumination DMD is fed by a single wavelength (of $740$ nm) from the Acousto-Optic Tunable Filter (AOTF) connected to a Supercontinnum (SuperK-EXR20) laser. The detection DMD applies the detection patterns and feeds the transmitted light to a 16-channel Spectro-photometer (PML-16C) which connects to a computer, containing the associated data-acquisition cards and software (SPC-150, DCC-100, and PCIe) for detecting and recording the data.

Fig. 6.

(a) Reconstruction results for the first experimental phantom having three thin capillaries embedded at a depth of $8.5$ mm from the plane of illumination. Each embedding has a different $\delta {\mu }_a$ value ( $0.004$ $m{m}^{-1}$ , $0.008$ $m{m}^{-1}$ , and $0.012$ $m{m}^{-1}$ ). (b) The results for the second experimental phantom with two of the tubes placed at depths of $6$ mm and $16$ mm from the plane of illumination. The middle tube is slanted and extends from $6$ mm to $16$ mm. The two tubes at the sides have a $\delta {\mu }_a$ value of $0.004$ $m{m}^{-1}$ , while the middle tube has $\delta {\mu }_a$ value of $0.008$ $m{m}^{-1}$ . The results are shown in terms of the reconstructed iso-volumes, averaged $\delta {\mu }_a$ distribution over the 2D planes, and quantitatively in terms of the MSE and the VE. The The results for LSQ, ModAM, and Z-AUTOMAP are produced at an iso-volume of $30\mathrm{\%}$ of the maximum value. The black lines represent the GT $\delta {\mu }_a$ values for each tube. The segmented micro-CT volumes are also shown in both cases.

Fig. 7.

Reconstruction results for the first experimental phantom having three thin capillaries embedded at a depth of $8.5$ mm from the plane of illumination. Each embedding has a different $\delta {\mu }_a$ value ( $0.004$ $m{m}^{-1}$ , $0.008$ $m{m}^{-1}$ , and $0.012$ $m{m}^{-1}$ ), obtained using a Laplacian-guided regularized LSQ-based technique. As before, the 3D reconstruction results are obtained using an iso-volume of 30% of the maximum value.