针对可见光和SAR图像由于成像机理不同导致图像内容差异大，深度特征难对齐，关联速度慢，提出一种深度多源哈希网络模型实现SAR图像和可见光图像间的关联。首先，针对SAR与光学遥感图像颜色信息差异大，提出图像变换机制，将光学图像转换生成4种不同类型的光谱图像输入到网络中，打乱颜色通道，让网络更加关注于图像的纹理和轮廓信息，而对颜色信息不敏感；其次，针对SAR图像噪声大，同一场景下2种模态图像内容异构，提出图像对训练策略，减小多源图像间的特征差异；然后，针对关联效率低，存储消耗大，提出三元组哈希损失函数，提升模型的关联准确率，降低关联时间；最后，构建了一个SAR与光学双模态遥感数据集SODMRSID，实验部分验证了数据集的实用性，同时提出算法的关联准确率明显优于现有算法。
Due to the different imaging mechanism of visible and SAR images， the content of images is different， the depth features are difficult to align， and the correlation speed is slow. A depth cross modal hash network model is proposed to realize the cross modal correlation between SAR images and optical images. Firstly， aiming at the great difference of color information between SAR and optical remote sensing images， an image transformation mechanism is proposed. Four different types of spectral images are generated from optical image conversion and input into the network， which disrupts the color channel， so that the network pays more attention to the texture and contour information of the image， but is not sensitive to the color information； secondly， aiming at the high noise of SAR image， two modal images in the same scene are generated. For example， the content is heterogeneous， image pair training strategy is proposed to reduce the feature difference between cross-modal images； then， aiming at the low correlation efficiency and high storage consumption， a triple hash loss function is proposed to improve the association accuracy of the model and reduce the association time. Finally， a SAR and optical dual-mode remote sensing data set is constructed. The experimental part verifies the practicability of the data set and the effectiveness of the proposed algorithm.
[1] LI B, XIE X Y, WEI X X, et al. Ship detection and classification from optical remote sensing images: A survey[J]. Chinese Journal of Aeronautics, 2021, 34(3): 145-163.
[2] LI D P. A novel method for multi-angle SAR image matching[J]. Chinese Journal of Aeronautics, 2015, 28(1): 240-249.
[3] WANG L W, LI Y, HUANG J, et al. Learning two-branch neural networks for image-text matching tasks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2): 394-407.
[4] YANG M, ZHAO W, XU W, et al. Multitask learning for cross-domain image captioning[J]. IEEE Transactions on Multimedia, 2019, 21(4): 1047-1061.
[5] MIN X, ZHAI G, ZHOU J, et al. A multimodal saliency model for videos with high audio-visual correspondence[J]. IEEE Transactions on Image Processing, 2020, 29(1): 3805-3819.
[6] PAREKH S, ESSID S, OZEROV A, et al. Weakly supervised representation learning for audio-visual scene analysis[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 416-428.
[7] XIANG X Z, LV N, YU Z T, et al. Cross-modality person re-identification based on dual-path multi-branch network[J]. IEEE Sensors Journal, 2019, 19(23): 11706-11713.
[8]
[9] LI Y S, ZHANG Y J, HUANG X, et al. Learning source-invariant deep hashing convolutional neural networks for cross-source remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(11): 6521-6536.
[10] XIONG W, XIONG Z Y, CUI Y Q, et al. A discriminative distillation network for cross-source remote sensing image retrieval[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 1234-1247.
[11] XIONG W, LV Y F, ZHANG X H, et al. Learning to translate for cross-source remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(7): 4860-4874.
[12]
[13]
[14] LU X Q, WANG B Q, ZHENG X T, et al. Exploring models and data for remote sensing image caption generation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(4): 2183-2195.
[15] SCHMITT M, HUGHES L H, ZHU X X. The sen1-2 dataset for deep learning in sar-optical data fusion[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 141-146.
[16] TORRES R, SNOEIJ P, GEUDTNER D, et al. GMES sentinel-1 mission[J]. Remote Sensing of Environment, 2012, 120: 9-24.
[17] DRUSCH M, BELLO U D, CARLIER S, et al. Sentinel-2: ESA'?s optical high-resolution mission for GMES operational services[J]. Remote Sensing of Environment, 2012, 120: 25-36.
[18]
[19] XU X, SHEN F M, YANG Y, et al. Learning discriminative binary codes for large-scale cross-modal retrieval[J]. IEEE Transactions on Image Processing, 2017, 26(5): 2494-2507.
[20]
CJA