基于MatchNet和多点匹配约束的可见光-SAR图像匹配
收稿日期: 2023-06-13
修回日期: 2023-07-17
录用日期: 2023-08-11
网络出版日期: 2023-09-01
基金资助
国家自然科学基金(61801491)
Optical⁃SAR image matching based on MatchNet and multi⁃point matching constraint
Received date: 2023-06-13
Revised date: 2023-07-17
Accepted date: 2023-08-11
Online published: 2023-09-01
Supported by
National Natural Science Foundation of China(61801491)
基于可见光和SAR成像的异源图像匹配是实现复杂场景下飞行器视觉导航的关键。由于成像机理的差异，可见光-SAR图像匹配需要面对成像几何特性差异、非线性辐射失真以及噪声干扰等挑战。近年来，基于深度学习的图像匹配方法表现出比传统方法更优的场景适应性，但现有深度学习方法仍未完全解决上述几个挑战，还无法完全满足视觉导航中的高精度匹配需求。基于经典的MatchNet匹配框架，针对其单个模板匹配精度不足的问题，提出一种多点匹配约束的匹配方法（MMC-MatchNet）。方法主要分为2步：第1步生成子模板并利用MatchNet进行匹配，第2步基于多点几何约束剔除误匹配子模板并优化匹配结果。该方法在保持MatchNet输入尺寸不变的同时提升了网络对多尺寸可见光-SAR图像的匹配精度。此外，还提出了一种多级数据集构建方法来训练MatchNet。与随机采样构建负样本的方式相比，所提方法更关注于匹配任务中的困难样本，是一种适用于图像匹配的数据增强手段。对MMC-MatchNet在2种可见光-SAR图像匹配数据上进行了测试，取得了比NCC、NMI、HOPC以及单模板MatchNet方法更好的匹配结果，在与HOPC匹配精度相似的前提下，MMC-MatchNet可大幅提高匹配正确率（CMR）。同时，多点匹配约束和多级数据构建方法还适用于其他基于Patch的图像匹配模型。
叶熠彬 , 滕锡超 , 于起峰 , 李璋 . 基于MatchNet和多点匹配约束的可见光-SAR图像匹配[J]. 航空学报, 2024 , 45(10) : 329162 -329162 . DOI: 10.7527/S1000-6893.2023.29162
Multimodal image matching between optical and SAR images is critical for visual navigation of the aircraft flying over complex areas. Due to different imaging mechanisms， accurate optical-SAR image matching faces several challenges， such as imaging geometry differences between optical and SAR， nonlinear radiation distortion of SAR， and noise interference. Although deep learning-based image matching methods have shown better adaptability than traditional methods， they still cannot fully solve the above-mentioned challenges and their matching accuracy may be not enough for visual navigation tasks. This paper proposes a new image matching method called Multi-point Matching Constraint MatchNet （MMC-MatchNet） based on the typical MatchNet framework， so as to improve the matching accuracy of the single template matching using MatchNet. The method consists of two main steps： sub-templates sampling and matching using MatchNet， and mismatched sub-templates removal based on multi-point geometric constraint. The original input image size of the MatchNet is maintained， while the matching accuracy for optical-SAR image matching is improved. This paper also proposes a multi-level training dataset generation method to train the MMC-MatchNet. Compared to the random sampling method， our method focuses on the difficult samples in image matching， and can be seen as a special data augmentation method for image matching. MMC-MatchNet is tested on two multi-modal image matching datasets， outperforming NCC， NMI， HOPC and the single-template MatchNet. With similar matching accuracy to that of HOPC， MMC-MatchNet can improve the Correctly Matching Rate （CMR）. The method based on multi-point matching constraint and multi-level dataset generation can be easily extend to other patch-based image matching models.
Key words： image matching; template matching; optical image; SAR image; MatchNet
| 1 | 刘中杰， 曹云峰， 庄丽葵， 等. 基于控制线方法的机载SAR和可见光图像匹配应用研究［J］. 航空学报， 2013， 34（9）： 2194-2201. |
| LIU Z J， CAO Y F， ZHUANG L K， et al. Applied research on airborne SAR and optical image registration based on control line method［J］. Acta Aeronautica et Astronautica Sinica， 2013， 34（9）： 2194-2201 （in Chinese）. | |
| 2 | 赵耀， 熊智， 田世伟， 等. 基于SAR图像匹配结果可信度评价的INS/SAR自适应Kalman滤波算法［J］. 航空学报， 2019， 40（8）： 322850. |
| ZHAO Y， XIONG Z， TIAN S W， et al. INS/SAR adaptive Kalman filtering algorithm based on credibility evaluation of SAR image matching results［J］. Acta Aeronautica et Astronautica Sinica， 2019， 40（8）： 322850 （in Chinese）. | |
| 3 | LU J Z， HU M Q， DONG J， et al. A novel dense descriptor based on structure tensor voting for multi-modal image matching［J］. Chinese Journal of Aeronautics， 2020， 33（9）： 2408-2419. |
| 4 | JIANG X Y， MA J Y， XIAO G B， et al. A review of multimodal image matching： Methods and applications［J］. Information Fusion， 2021， 73（C）： 22-71. |
| 5 | LI Q， FU B H， DONG Y F. Registration of radar and optical satellite images using multiscale filter technique and information measure［M］∥ IMPERATORE P， RICCIO D， eds. Geoscience and Remote Sensing New Achievements. London： InTech， 2010. |
| 6 | HONG T D， SCHOWENGERDT R A. A robust technique for precise registration of radar and optical satellite images［J］. Photogrammetric Engineering & Remote Sensing， 2005， 71（5）： 585-593. |
| 7 | PAUL S， PATI U C. A comprehensive review on remote sensing image registration［J］. International Journal of Remote Sensing， 2021， 42（14）： 5400-5436. |
| 8 | ZHAO F， HUANG Q M， WEN G. Image matching by normalized cross-correlation［C］∥ 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings. Piscataway： IEEE Press， 2006： Ⅱ. |
| 9 | WELLS W M， VIOLA P， ATSUMI H， et al. Multi-modal volume registration by maximization of mutual information［J］. Medical Image Analysis， 1996， 1（1）： 35-51. |
| 10 | YE Y X， SHAN J， BRUZZONE L， et al. Robust registration of multimodal remote sensing images based on structural similarity［J］. IEEE Transactions on Geoscience and Remote Sensing， 2017， 55（5）： 2941-2958. |
| 11 | SURI S， REINARTZ P. Mutual-information-based registration of TerraSAR-X and ikonos imagery in urban areas［J］. IEEE Transactions on Geoscience and Remote Sensing， 2010， 48（2）： 939-949. |
| 12 | YE Y X， SHEN L， HAO M， et al. Robust optical-to-SAR image matching based on shape properties［J］. IEEE Geoscience and Remote Sensing Letters， 2017， 14（4）： 564-568. |
| 13 | FAN B， HUO C L， PAN C H， et al. Registration of optical and SAR satellite images by exploring the spatial relationship of the improved SIFT［J］. IEEE Geoscience and Remote Sensing Letters， 2013， 10（4）： 657-661. |
| 14 | HUANG L， LI Z. Feature-based image registration using the shape context［J］. International Journal of Remote Sensing， 2010， 31（8）： 2169-2177. |
| 15 | XIANG Y M， WANG F， YOU H J. OS-SIFT： A robust SIFT-like algorithm for high-resolution optical-to-SAR image registration in suburban areas［J］. IEEE Transactions on Geoscience and Remote Sensing， 2018， 56（6）： 3078-3090. |
| 16 | WANG Y G， YU X D， ZHANG Y， et al. An adaptive SAR and optical images registration approach based on SOI-SIFT［C］∥ IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2022： 2582-2585. |
| 17 | ZHANG X T， WANG Y H， LIU H W. Robust optical and SAR image registration based on OS-SIFT and cascaded sample consensus［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 19： 4011605. |
| 18 | FAN J W， YE Y X， LIU G C， et al. Phase congruency order-based local structural feature for SAR and optical image matching［J］. IEEE Geoscience and Remote Sensing Letters， 2022， 19： 4507105. |
| 19 | RAGURAM R， CHUM O， POLLEFEYS M， et al. USAC： A universal framework for random sample consensus［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（8）： 2022-2038. |
| 20 | ZHAO M， ZHANG G X， DING M. Heterogeneous self-supervised interest point matching for multi-modal remote sensing image registration［J］. International Journal of Remote Sensing， 2022， 43（3）： 915-931. |
| 21 | XIANG D L， XIE Y Z， CHENG J D， et al. Optical and SAR image registration based on feature decoupling network［J］. IEEE Transactions on Geoscience and Remote Sensing， 1858， 60： 5235913. |
| 22 | LIPMAN Y， YAGEV S， PORANNE R， et al. Feature matching with bounded distortion［J］. ACM Transactions on Graphics， 2014， 33（3）： 26. |
| 23 | MA J Y， ZHAO J， JIANG J J， et al. Locality preserving matching［J］. International Journal of Computer Vision， 2019， 127（5）： 512-531. |
| 24 | BIAN J W， LIN W Y， MATSUSHITA Y， et al. GMS： Grid-based motion statistics for fast， ultra-robust feature correspondence［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 2828-2837. |
| 25 | VERDIE Y， YI K M， FUA P， et al. TILDE： A temporally invariant learned DEtector［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 5279-5288. |
| 26 | ZHANG X， YU F X， KARAMAN S， et al. Learning discriminative and transformation covariant local feature detectors［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 4923-4931. |
| 27 | LUO Z X， SHEN T W， ZHOU L， et al. ContextDesc： Local descriptor augmentation with cross-modality context［C］∥ 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2019： 2522-2531. |
| 28 | YI K M， TRULLS E， LEPETIT V， et al. LIFT： Learned invariant feature transform［C］∥ European Conference on Computer Vision. Cham： Springer， 2016： 467-483. |
| 29 | WANG J， ZHOU F， WEN S L， et al. Deep metric learning with angular loss［C］∥ 2017 IEEE International Conference on Computer Vision （ICCV）. Piscataway： IEEE Press， 2017： 2612-2620. |
| 30 | HAN X F， LEUNG T， JIA Y Q， et al. MatchNet： Unifying feature and metric learning for patch-based matching［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 3279-3286. |
| 31 | ZAGORUYKO S， KOMODAKIS N. Learning to compare image patches via convolutional neural networks［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 4353-4361. |
| 32 | MA W P， ZHANG J， WU Y， et al. A novel two-step registration method for remote sensing images based on deep and local features［J］. IEEE Transactions on Geoscience and Remote Sensing， 2019， 57（7）： 4834-4843. |
| 33 | HUGHES L H， SCHMITT M， MOU L C， et al. Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN［J］. IEEE Geoscience and Remote Sensing Letters， 2018， 15（5）： 784-788. |
| 34 | HOFFMANN S， BRUST C A， SHADAYDEH M， et al. Registration of high resolution SAR and optical satellite imagery using fully convolutional networks［C］∥ IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2019： 5152-5155. |
| 35 | HUGHES L H， MARCOS D， LOBRY S， et al. A deep learning framework for matching of SAR and optical imagery［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2020， 169： 166-179. |
| 36 | QUAN D， WANG S， LIANG X F， et al. Deep generative matching network for optical and SAR image registration［C］∥ IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2018： 6215-6218. |
| 37 | MERKLE N， AUER S， MüLLER R， et al. Exploring the potential of conditional adversarial networks for optical and SAR image matching［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2018， 11（6）： 1811-1820. |
| 38 | BALNTAS V， RIBA E， PONSA D， et al. Learning local feature descriptors with triplets and shallow convolutional neural networks［C］∥ Proceedings of the British Machine Vision Conference 2016， 2016. |
| 39 | MISHCHUK A， MISHKIN D， RADENOVI? F， et al. Working hard to know your neighbor’s margins： Local descriptor learning loss［C］∥ Proceedings of the 31st International Conference on Neural Information Processing Systems. New York： ACM， 2017： 4829–4840. |
| 40 | TIAN Y R， FAN B， WU F C. L2-net： Deep learning of discriminative patch descriptor in euclidean space［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 6128-6136. |
| 41 | HE H Q， CHEN M， CHEN T， et al. Matching of remote sensing images with complex background variations via Siamese convolutional neural network［J］. Remote Sensing， 2018， 10（3）： 355. |
| 42 | HE H Q， CHEN M， CHEN T， et al. Learning to match multitemporal optical satellite images using multi-support-patches Siamese networks［J］. Remote Sensing Letters， 2019， 10（6）： 516-525. |
| 43 | LIAO Y， DI Y D， ZHOU H， et al. Feature matching and position matching between optical and SAR with local deep feature descriptor［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2021， 15： 448-462. |
| 44 | QUAN D， WEI H Y， WANG S， et al. Self-distillation feature learning network for optical and SAR image registration［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 4706718. |
| 45 | ZHU R J， YU D W， JI S P， et al. Matching RGB and infrared remote sensing images with densely-connected convolutional neural networks［J］. Remote Sensing， 2019， 11（23）： 2836. |
| 46 | MA J Y， JIANG X Y， FAN A X， et al. Image matching from handcrafted to deep features： A survey［J］. International Journal of Computer Vision， 2021， 129（1）： 23-79. |
| 47 | ZHANG H， LEI L， NI W P， et al. Explore better network framework for high-resolution optical and SAR image matching［J］. IEEE Transactions on Geoscience and Remote Sensing， 2021， 60： 4704418. |
| 48 | YE Y X， TANG T F， ZHU B， et al. A multiscale framework with unsupervised learning for remote sensing image registration［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5622215. |
| 49 | FANG Y Y， HU J， DU C， et al. SAR-optical image matching by integrating Siamese U-net with FFT correlation［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 19： 4016505. |
| 50 | LIN T Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］∥ 2017 IEEE International Conference on Computer Vision （ICCV）. Piscataway： IEEE Press， 2017： 2999-3007. |
| 51 | ZHENG L X， XIAO G B， SHI Z W， et al. MSA-net： Establishing reliable correspondences by multiscale attention network［J］. IEEE Transactions on Image Processing： a Publication of the IEEE Signal Processing Society， 2022， 31： 4598-4608. |
/
| 〈 |
|
〉 |
CJA