Semantic redundancy-aware implicit neural compression for multidimensional biomedical image data

Abstract

The surge in advanced imaging techniques has generated vast biomedical image data with diverse dimensions in space, time and spectrum, posing big challenges to conventional compression techniques in image storage, transmission, and sharing. Here, we propose an intelligent image compression approach with the first-proved semantic redundancy of biomedical data in the implicit neural function domain. This Semantic redundancy based Implicit Neural Compression guided with Saliency map (SINCS) can notably improve the compression efficiency for arbitrary-dimensional image data in terms of compression ratio and fidelity. Moreover, with weight transfer and residual entropy coding strategies, it shows improved compression speed while maintaining high quality. SINCS yields high quality compression with over 2000-fold compression ratio on 2D, 2D-T, 3D, 4D biomedical images of diverse targets ranging from single virus to entire human organs, and ensures reliable downstream tasks, such as object segmentation and quantitative analyses, to be conducted at high efficiency.

Fig. 1. Validation of semantic redundancy in implicit neural function domain correlation.

a Comparison of semantic correlations in three image modes. 2D, 3D and 4D (3D + T) images of the same zebrafish embryo heart were used as target, to evaluate the lateral (mode 1, top), axial (mode 2, middle), and temporal (mode 3, bottom) semantic correlations, respectively. In each mode, the images’ semantic correlations in the implicit neural function domain and spatial domain are compared through calculating their parameter histograms. The Average Wilson Coefficient (AWC) values are used as correlation metric with lower value indicating higher correlation. b Clusterings for structurally-similar data in implicit neural function and spatial domain. Blood vessels and cell nuclei are chosen to represent line-like and point-like signals, respectively. The Rayleigh Entropy (RE) values are calculated to quantify the clusterings, with lower value indicating more compact clustering of the images. Scale bars from top to bottom: 5 μm, 10 μm. c Comparison of image correlations in implicit neural function domain (AWC metric), and in spatial domain (SSIM metric). Multi-scale samples captured by different imaging techniques are compared to validate the universally-high correlations in implicit neural function domain. The human skeleton, human brain images are obtained from CT (Multi-Slice Spiral CT, Medium slices with 2.5 mm thickness) and MRI (T1-weighted MRI), respectively; the 3D images of zebrafish embryo heart and mouse brain neurons are obtained by light-sheet microscope (4×/0.13 NA illumination and 20×/0.5 NA detection for heart, 4×/0.28 NA illumination and 10×/0.3 NA detection for mouse brain neurons); the 2D subcellular images of cell nuclei and mitochondrial are captured by light-sheet microscope (20×/0.45 NA detection objective for cell nuclei and 60×/1.1 NA detection objective for mitochondrial), and the 4D subcellular images of dynamics microtubes are obtained from single objective light-sheet microscope (100×/1.5 NA for illumination and detection); the virus images are obtained from electron microscopy (an LEO (Zeiss, Oberkochen, Germany) with a Morada (Olympus) camera). Scale bars from left to right: 5 cm, 5 cm, 20 μm, 10 μm, 10 μm, 2 μm, 1 μm, 10 nm.

Fig. 2. The workflow of SINCS compression and decompression.

a Compression pipeline of SINCS. For a given large-scale biomedical dataset, the pipelie contains: (i) the data is first effectively partitioned into several groups through an adaptive grouping strategy; (ii) saliency mechanism was introduced to realize adaptive compression fitting for biomedical data. This mechanism leverages saliency hot maps (serve as discrete probability distributions for coordinates query) to optimize the compression process,enabling targeted learning of crucial information in the dataset; (iii) Multi-Layer Perceptron (MLP) was constructed as a parameterized mapping function to fit each group data. After saliency-guided sampling, the selected coordinates are first encoded to vectors with high-frequency by positional encoding, and then fed into the MLP,achieving “data-function(weights)” encoding. Subsequently, a weight transfer with trianed parameters θ₁ of 1^st group data as the initial parameters and θ₂ of 2nd group data for starting optimization was applied to promote network’s rapid convergence based on the high correlation between groups; (iv) after global fitting of the original data, a higher compression ratio can be further realized using weight-residual entropy coding strategy. Specifically, SINCS encoded residuals by subtracting the network parameters between neighboring networks and applying entropy coding to obtain the encoded initial parameters and residual ones. This step produces a bitstream at last. b Decompression pipeline of SINCS. In the decompression process, the weight-residual entropy decoding is adopted to convert the bitstream to original initial parameters θ₁ and remaining residual parameters Δθ_i. Subsequently, by successively adding the residuals to the initial weight parameters, the original weight parameters are obtained for each network corresponding to each group data. By modeling forward inference and regrouping, the decompressed data can be reordered by the pre-defined grouping strategy. c The decompression procedure showing that the region-specific decompression and visualization can be readily achieved in SINCS with flexibility, owing to its coordinate-based representation.

Fig. 3. Demonstration of SINCS compression capability on 2D bright field / 3D fluorescence images and performance on downstream tasks.

a Brief illustration of 2D data patch grouping and 3D data block grouping (The respective saliency maps are shown in the bottom right corner). Scale bars from top to bottom: 100 μm, 40 μm. b Comparison of bright-field 2D cell data and 3D cell nuclei data (labeled by GFP) reconstructed by different image compression methods. Since JPEG cannot compress 3D data, we convert 3D to 2D data for batch compression, where the data compression ratio is 130× for JPEG (limitation) and 580× for other methods. Scale bars from top to bottom: 5 μm, 5 μm. c Overall performance rating using PSNR and SSIM metrics, to show that SINCS surpass alternative compression approaches in terms of higher structural fidelity. In the box plots in c, the line within each box represents the mean; the outer edges of the box are the 10th and 90th percentiles; and the whiskers extend to the minimum and maximum values. d Comparative segmentations of 2D cells and intersection over union (IoU) scores of 3D cell nuclei by different compression methods. The IoU scores are used as fidelity metric with higher score indicating higher visual fidelty. TP (True Positive): the correctly reconstructed structures; FP (FalsePositive): the incorrectly hallucinated structures; FN (False Negative): the missing details. Metrics from top to bottom: Counting accuracy, and IoU score comparision. Scale bars from top to bottom: 5 μm, 5 μm. e Histograms comparing the reconstruction accuracy of different compression methods with using 2D counting accuracy (top) and 3D IoU scores as metric (bottom).

Fig. 4. High intensity fidelity SINCS compression on sequential Ca²⁺ images of moving C. elegans allowing downstream quantification of neural activities.

a The motor neurons in an entire L4 C. elegans larva reconstructed by different compression approaches (The top right corner shows the C. elegans crawling trend with a time-coded trace). The magnified views of indicated regions show that H.265 (1430×) and INR-S (1500×) lose a considerable amount of weak signals, owing to the high signal dynamic range. In sharp contrast, SINCS preserves these weak signals perfectly. Meanwhile, SINCS reconstruction also shows spatial resolution higher than the other two approachs, notably contributing to the resolving of dense signals. The SSIM are used as structral fidelity metric with higher value indicating higher fidelity. Scale bar: 10 μm. b Spatio-termporal patterns of 4 motor neurons (VB1, VB4, VB8, VA7) reconstructed by different image compression methods. The neuron tracing trajectories are displayed on the left, indicating the dynamics of neuronal signals in spatial domain. The Ca²⁺ activity curves of corresponding neurons reconstructed by SINCS (red), INR-S (green) and H.265 (gray) approaches are shown on the right, and compared with the ground truth curve plotted by raw data (yellow). The intensity correlations are used as metrics to quantify the intensity fidelity of the reconstructions by diverse methods, with a higher correlation value indicating a higher stability in intensity fidelity. c Overall performance rating using PSNR (top), signal preservation (middle) and intensity correlation (bottom) metrics, to show that SINCS surpass the H.265 and INR-S in term of both high structural and intensity fidelities. In the box plots in c, the line within each box represents the mean; the outer edges of the box are the 10th and 90th percentiles; and the whiskers extend to the minimum and maximum values.

Fig. 5. SINCS compression on 4D super-resolution images of mitochondrial dynamics and 5D light-sheet images of CAR-T cell / tumor cell interaction.

a 3D volume renderings of GT (top) and 700× SINCS compression result (bottom) showing the overall high structural similarity by SINCS. Scale bar: 10 μm. b Comparison of mitochondrion fission process reconstructed by H.265-S (678×), INR-S (700×) and SINCS (700×).The red arrows indicate the mitochondrion fission site over 4 s. As compared to the GT from raw images, only SINCS results are capable of resolving the fine structural changes. The SSIM scores of the reconstructions by three approaches are calculated, with higher value indicating higher fidelity. Scale bar: 1 μm. c Comparative results of mitochondrion morphological changes in reconstructed cross section plane. The cross sention plane at different time points demonstrate the mitochondrion contraction and expansion at nanoscale. The IoU scores are used as fidelity metric with higher score indicating higher reconstruction fidelty during the morphological and cross-sectional area changes. TP (True Positive): the correctly reconstructed structures; FP (FalsePositive): the incorrectly hallucinated structures; FN (False Negative): the missing details. Scale bar: 1 μm. d Quantitative comparison of reconstruction accuracy at mitochondrial fission site with using European distance as metric (top) and cross-section plane using area as metric (bottom). e Visual comparison of GT and 2302× SINCS compression result of 5D light-sheet fluorescence microscopy data recording the interactions between CAR-T (labeled by GFP) and Nalm6 tumor cells (labeled by Dsred) in 20 min. Scale bar: 5 μm. f Comparison of synaptic area variation and SSIM values between GT and SINCS result. In the box plots in d and f, the line within each box represents the mean; the outer edges of the box are the 10th and 90th percentiles; and the whiskers extend to the minimum and maximum values.