航空学报 > 2022, Vol. 43 Issue (S1): 726938-726938   doi: 10.7527/S1000-6893.2022.26938

基于无监督深度学习的红外与可见光图像融合

孙秀一1,2, 胡绍海1,2, 马晓乐1,2   

  1. 1. 北京交通大学 信息科学研究所,北京 100044;
    2. 现代信息科学与网络技术北京市重点实验室,北京 100044
  • 收稿日期:2022-01-11 修回日期:2022-01-14 发布日期:2022-04-06
  • 通讯作者: 胡绍海,E-mail:shhu@bjtu.edu.cn E-mail:shhu@bjtu.edu.cn
  • 基金资助:
    国家自然科学基金 (62172030,61771058);中央高校基础研究基金(2021JBM009)

Infrared and visible image fusion based on unsupervised deep learning

SUN Xiuyi1,2, HU Shaohai1,2, MA Xiaole1,2   

  1. 1. Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China;
    2. Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing 100044, China
  • Received:2022-01-11 Revised:2022-01-14 Published:2022-04-06
  • Supported by:
    National Natural Science Foundation of China (62172030, 61771058), the Fundamental Research Funds for the Central Universities (2021JBM009)

摘要: 目前已知的多数基于卷积神经网络的红外与可见光图像融合模型没有充分利用来自可见光源图像的层次特征,因此导致融合图像细节纹理不足。受残差网络和密集网络的启发,提出了一种基于无监督深度学习的图像融合算法来解决融合图像细节纹理信息不足的问题。使用的残差密集块有连续存储机制,最大程度地保留每层的特征信息,局部残差融合和全局残差融合的设计有利于学习图像中的结构纹理。此外,为了更好地保留可见光图像中的细节纹理,引入了生成对抗网络对数据集进行无监督学习。主客观实验表明,该算法不仅获得了良好的视觉融合效果,融合图像具有更多的边缘纹理信息,在客观评价指标上对比现有优秀的算法也较大的提升。

关键词: 无监督学习, 残差密集网络, 生成对抗网络, 红外图像, 可见光图像, 图像融合

Abstract: Most of the known infrared and visible image fusion models based on convolutional neural networks make little use of the hierarchical features from visible images, thus resulting in insufficient texture details of the fused image. Inspired by the residual network and dense network, an image fusion algorithm is proposed based on unsupervised deep learning to solve the problem of insufficient texture information of fused images. The residual dense block has a continuous storage mechanism to retain the feature information of each layer to the maximum extent. The design of local residual fusion and global residual fusion is conducive to learning the structural texture in the image. In addition, to better preserve the detailed texture in visible images, the generative adversarial network is introduced to perform unsupervised learning on the dataset. Subjective and objective experiments show that the proposed algorithm achieves not only a good visual fusion effect, but also more edge texture information of the fused image. Compared with that of the existing state-of-the-art algorithms, the objective evaluation index of the method proposed is also greatly improved.

Key words: unsupervised learning, residual dense network, generative adversarial network, infrared image, visible image, image fusion

中图分类号: