Efficient Photoacoustic Image Synthesis with Deep Learning

Abstract

Photoacoustic imaging potentially allows for the real-time visualization of functional human tissue parameters such as oxygenation but is subject to a challenging underlying quantification problem. While in silico studies have revealed the great potential of deep learning (DL) methodology in solving this problem, the inherent lack of an efficient gold standard method for model training and validation remains a grand challenge. This work investigates whether DL can be leveraged to accurately and efficiently simulate photon propagation in biological tissue, enabling photoacoustic image synthesis. Our approach is based on estimating the initial pressure distribution of the photoacoustic waves from the underlying optical properties using a back-propagatable neural network trained on synthetic data. In proof-of-concept studies, we validated the performance of two complementary neural network architectures, namely a conventional U-Net-like model and a Fourier Neural Operator (FNO) network. Our in silico validation on multispectral human forearm images shows that DL methods can speed up image generation by a factor of 100 when compared to Monte Carlo simulations with 5×108 photons. While the FNO is slightly more accurate than the U-Net, when compared to Monte Carlo simulations performed with a reduced number of photons (5×106), both neural network architectures achieve equivalent accuracy. In contrast to Monte Carlo simulations, the proposed DL models can be used as inherently differentiable surrogate models in the photoacoustic image synthesis pipeline, allowing for back-propagation of the synthesis error and gradient-based optimization over the entire pipeline. Due to their efficiency, they have the potential to enable large-scale training data generation that can expedite the clinical application of photoacoustic imaging.

Keywords: Fourier Neural Operator; Monte Carlo simulation; deep learning; image synthesis; multispectral functional imaging; photoacoustic imaging; surrogate model.

Figure A1

Performance of the Fourier Neural Operator (FNO) and the U-Net when using the Monte Carlo eXtreme (MCX) reference initial pressure distributions. According to the sample-wise max-normalized peak signal-to-noise ratio (top) and structural similarity index measure (bottom), the FNO consistently slightly outperforms the U-Net on the 220 test cases.

Figure 1

Our deep learning-based approach to simulating photon transport in biological tissue replaces slow state-of-the-art Monte Carlo simulations in the photoacoustic image synthesis pipeline.

Figure 2

Neural networks are trained on synthetic data. (1) Image synthesis is based on a literature-based anatomy model (here: of the human forearm) and comprises the steps of tissue geometry generation, optical parameter assignment (absorption coefficient ${\mu }_a$, scattering coefficient ${\mu }_s$ and scattering anisotropy g), and simulation of the initial pressure distribution $p_0$. (2) Neural networks are trained with pairs of initial pressure distribution and corresponding parameter maps, here with either a modified U-Net with three input channels or a Fourier Neural Operator (FNO) network.

Figure 3

Performance of the Fourier Neural Operator (FNO) and the U-Net when using the Monte Carlo eXtreme (MCX) reference initial pressure distributions. According to the sample-wise max-normalized mean absolute error (MAE), the FNO consistently slightly outperforms the U-Net on the 220 test cases.

Figure 4

Results of initial pressure distribution $p_0$ estimation for the Fourier Neural Operator (FNO) $\widehat{p_0}$. Best, median, and worst (top to bottom) performing samples for the FNO according to the normalized mean absolute error (MAE) on 800 $\mathrm{n}$$\mathrm{m}$. Most misestimations occur in high-absorbing structures such as vessels or skin. Poorly performing samples mostly feature thicker skin or high-absorbing vessels close to the skin.

Figure 5

Synthesis accuracy measured with the max-normalized mean absolute error (MAE) as a function of the number of training cases for Fourier Neural Operator (FNO) network (left) and the number of photons for Monte Carlo eXtreme (MCX) simulations (right). Note that the reference simulations, which were used to compute the mean absolute error (MAE), were performed with 5 × 10⁸ photons.

Figure 6

Estimation errors of Fourier Neural Operator (FNO) network and simulation errors of Monte Carlo eXtreme (MCX) of the same sample with fewer photons are different in nature. While deep learning estimations show errors at anatomical structure boundaries, MCX simulations with fewer photons contain general unstructured noise (apparent in the logarithmic visualization in the top row).

Figure 7

During inference, U-Net, and Fourier Neural Operator (FNO) networks estimate the initial pressure distribution two orders of magnitude faster than Monte Carlo simulations from Monte Carlo eXtreme (MCX) with representative configurations. Results were obtained by averaging the run time of the optical forward step of 22 samples and all 16 wavelengths, i.e., the reported run time is for one wavelength of a single sample. The error bars indicate the standard deviation.