Progressive decomposition of infrared and visible image fusion network with joint transformer and Resnet

Abstract

The objective of image fusion is to synthesize information from multiple source images into a single, high-quality composite that is information-rich, thereby enhancing both human visual interpretation and machine perception capabilities. This process also establishes a robust foundation for downstream image-related tasks. Nevertheless, current deep learning-based networks frequently neglect the distinctive features inherent in source images, presenting challenges in effectively balancing the interplay between basic and detailed features. To tackle this limitation, we introduce a progressive decomposition network that integrates Lite Transformer (LT) and ResNet architecture for infrared and visible image fusion (IVIF). Our methodology unfolds in three principal stages: Initially, a foundational convolutional neural network (CNN) is deployed to extract coarse-scale features from the source images. Subsequently, the LT is employed to bifurcate these coarse features into basic and detailed feature components. In the second phase, to augment the detail information across various inter-layer extractions, we substitute the conventional ResNet preprocessing with a combination of coarse and LT module. Cascade LT operations are implemented following the initial two ResNet blocks (ResB), enabling two-branch feature extraction from these reconfigured blocks. The final stage involves the design of specialized fusion sub-networks to process the basic and detail information blocks extracted from different layers. These processed image feature blocks are then channeled through semantic injection module (SIM) and Transformer decoders to generate the fused image. Complementing this architecture, we have developed a semantic information extraction module that aligns with the progressive inter-layer detail extraction framework. The LT module is strategically embedded within the ResNet network architecture to optimize the extraction of both basic and detailed features across diverse layers. Moreover, we introduce a novel correlation loss function that operates on the basic and detail information between layers, facilitating the correlation of basic features while maintaining the independence of detail features across layers. Through comprehensive qualitative and quantitative analyses conducted on multiple infrared-visible datasets, we demonstrate the superior potential of our proposed network for advanced visual tasks. Our network exhibits remarkable performance in detail extraction, significantly outperforming existing deep learning methodologies in this domain.

Fig 1. Provides a visual comparison of different fusion methods.

Fig 2. A comparative analysis of streamlined fusion frameworks.

Fig 3. Overall framework diagram.

Fig 4. Annotated supplementary documentation for the comprehensive framework schematic.

Fig 5. A practical demonstration of key modules.

Fig 6. IR & VIS image encoding flowchart.

Fig 7. IR & VIS image decoding flowchart.

Fig 8. Qualitative comparison of ten fusion methods on scene 05005 from the RoadScene dataset.

Fig 9. Localized zoom effects for ten fusion results of the 05005 scene.

Fig 10. Cumulative distribution of the six metrics on the RoadScene dataset.

Fig 11. Qualitative comparison of ten fusion methods on scene 00754N from the MSRS dataset.

Fig 12. Cumulative distribution of the six metrics on the MSRS dataset.

Fig 13. Qualitative comparison of ten fusion methods on scene 01443 from the M³FD dataset.

Fig 14. Cumulative distribution of the six metrics on the M³FD dataset.

Fig 15. Segmentation results of ten fusion methods under BANet.

Fig 16. Visualization results of ablation experiments.