ACTA AERONAUTICAET ASTRONAUTICA SINICA >
Optical⁃SAR image matching based on MatchNet and multi⁃point matching constraint
Received date: 2023-06-13
Revised date: 2023-07-17
Accepted date: 2023-08-11
Online published: 2023-09-01
Supported by
National Natural Science Foundation of China(61801491)
Multimodal image matching between optical and SAR images is critical for visual navigation of the aircraft flying over complex areas. Due to different imaging mechanisms， accurate optical-SAR image matching faces several challenges， such as imaging geometry differences between optical and SAR， nonlinear radiation distortion of SAR， and noise interference. Although deep learning-based image matching methods have shown better adaptability than traditional methods， they still cannot fully solve the above-mentioned challenges and their matching accuracy may be not enough for visual navigation tasks. This paper proposes a new image matching method called Multi-point Matching Constraint MatchNet （MMC-MatchNet） based on the typical MatchNet framework， so as to improve the matching accuracy of the single template matching using MatchNet. The method consists of two main steps： sub-templates sampling and matching using MatchNet， and mismatched sub-templates removal based on multi-point geometric constraint. The original input image size of the MatchNet is maintained， while the matching accuracy for optical-SAR image matching is improved. This paper also proposes a multi-level training dataset generation method to train the MMC-MatchNet. Compared to the random sampling method， our method focuses on the difficult samples in image matching， and can be seen as a special data augmentation method for image matching. MMC-MatchNet is tested on two multi-modal image matching datasets， outperforming NCC， NMI， HOPC and the single-template MatchNet. With similar matching accuracy to that of HOPC， MMC-MatchNet can improve the Correctly Matching Rate （CMR）. The method based on multi-point matching constraint and multi-level dataset generation can be easily extend to other patch-based image matching models.
Key words： image matching; template matching; optical image; SAR image; MatchNet
Yibin YE , Xichao TENG , Qifeng YU , Zhang LI . Optical⁃SAR image matching based on MatchNet and multi⁃point matching constraint[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2024 , 45(10) : 329162 -329162 . DOI: 10.7527/S1000-6893.2023.29162
| 1 | 刘中杰， 曹云峰， 庄丽葵， 等. 基于控制线方法的机载SAR和可见光图像匹配应用研究［J］. 航空学报， 2013， 34（9）： 2194-2201. |
| LIU Z J， CAO Y F， ZHUANG L K， et al. Applied research on airborne SAR and optical image registration based on control line method［J］. Acta Aeronautica et Astronautica Sinica， 2013， 34（9）： 2194-2201 （in Chinese）. | |
| 2 | 赵耀， 熊智， 田世伟， 等. 基于SAR图像匹配结果可信度评价的INS/SAR自适应Kalman滤波算法［J］. 航空学报， 2019， 40（8）： 322850. |
| ZHAO Y， XIONG Z， TIAN S W， et al. INS/SAR adaptive Kalman filtering algorithm based on credibility evaluation of SAR image matching results［J］. Acta Aeronautica et Astronautica Sinica， 2019， 40（8）： 322850 （in Chinese）. | |
| 3 | LU J Z， HU M Q， DONG J， et al. A novel dense descriptor based on structure tensor voting for multi-modal image matching［J］. Chinese Journal of Aeronautics， 2020， 33（9）： 2408-2419. |
| 4 | JIANG X Y， MA J Y， XIAO G B， et al. A review of multimodal image matching： Methods and applications［J］. Information Fusion， 2021， 73（C）： 22-71. |
| 5 | LI Q， FU B H， DONG Y F. Registration of radar and optical satellite images using multiscale filter technique and information measure［M］∥ IMPERATORE P， RICCIO D， eds. Geoscience and Remote Sensing New Achievements. London： InTech， 2010. |
| 6 | HONG T D， SCHOWENGERDT R A. A robust technique for precise registration of radar and optical satellite images［J］. Photogrammetric Engineering & Remote Sensing， 2005， 71（5）： 585-593. |
| 7 | PAUL S， PATI U C. A comprehensive review on remote sensing image registration［J］. International Journal of Remote Sensing， 2021， 42（14）： 5400-5436. |
| 8 | ZHAO F， HUANG Q M， WEN G. Image matching by normalized cross-correlation［C］∥ 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings. Piscataway： IEEE Press， 2006： Ⅱ. |
| 9 | WELLS W M， VIOLA P， ATSUMI H， et al. Multi-modal volume registration by maximization of mutual information［J］. Medical Image Analysis， 1996， 1（1）： 35-51. |
| 10 | YE Y X， SHAN J， BRUZZONE L， et al. Robust registration of multimodal remote sensing images based on structural similarity［J］. IEEE Transactions on Geoscience and Remote Sensing， 2017， 55（5）： 2941-2958. |
| 11 | SURI S， REINARTZ P. Mutual-information-based registration of TerraSAR-X and ikonos imagery in urban areas［J］. IEEE Transactions on Geoscience and Remote Sensing， 2010， 48（2）： 939-949. |
| 12 | YE Y X， SHEN L， HAO M， et al. Robust optical-to-SAR image matching based on shape properties［J］. IEEE Geoscience and Remote Sensing Letters， 2017， 14（4）： 564-568. |
| 13 | FAN B， HUO C L， PAN C H， et al. Registration of optical and SAR satellite images by exploring the spatial relationship of the improved SIFT［J］. IEEE Geoscience and Remote Sensing Letters， 2013， 10（4）： 657-661. |
| 14 | HUANG L， LI Z. Feature-based image registration using the shape context［J］. International Journal of Remote Sensing， 2010， 31（8）： 2169-2177. |
| 15 | XIANG Y M， WANG F， YOU H J. OS-SIFT： A robust SIFT-like algorithm for high-resolution optical-to-SAR image registration in suburban areas［J］. IEEE Transactions on Geoscience and Remote Sensing， 2018， 56（6）： 3078-3090. |
| 16 | WANG Y G， YU X D， ZHANG Y， et al. An adaptive SAR and optical images registration approach based on SOI-SIFT［C］∥ IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2022： 2582-2585. |
| 17 | ZHANG X T， WANG Y H， LIU H W. Robust optical and SAR image registration based on OS-SIFT and cascaded sample consensus［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 19： 4011605. |
| 18 | FAN J W， YE Y X， LIU G C， et al. Phase congruency order-based local structural feature for SAR and optical image matching［J］. IEEE Geoscience and Remote Sensing Letters， 2022， 19： 4507105. |
| 19 | RAGURAM R， CHUM O， POLLEFEYS M， et al. USAC： A universal framework for random sample consensus［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（8）： 2022-2038. |
| 20 | ZHAO M， ZHANG G X， DING M. Heterogeneous self-supervised interest point matching for multi-modal remote sensing image registration［J］. International Journal of Remote Sensing， 2022， 43（3）： 915-931. |
| 21 | XIANG D L， XIE Y Z， CHENG J D， et al. Optical and SAR image registration based on feature decoupling network［J］. IEEE Transactions on Geoscience and Remote Sensing， 1858， 60： 5235913. |
| 22 | LIPMAN Y， YAGEV S， PORANNE R， et al. Feature matching with bounded distortion［J］. ACM Transactions on Graphics， 2014， 33（3）： 26. |
| 23 | MA J Y， ZHAO J， JIANG J J， et al. Locality preserving matching［J］. International Journal of Computer Vision， 2019， 127（5）： 512-531. |
| 24 | BIAN J W， LIN W Y， MATSUSHITA Y， et al. GMS： Grid-based motion statistics for fast， ultra-robust feature correspondence［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 2828-2837. |
| 25 | VERDIE Y， YI K M， FUA P， et al. TILDE： A temporally invariant learned DEtector［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 5279-5288. |
| 26 | ZHANG X， YU F X， KARAMAN S， et al. Learning discriminative and transformation covariant local feature detectors［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 4923-4931. |
| 27 | LUO Z X， SHEN T W， ZHOU L， et al. ContextDesc： Local descriptor augmentation with cross-modality context［C］∥ 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2019： 2522-2531. |
| 28 | YI K M， TRULLS E， LEPETIT V， et al. LIFT： Learned invariant feature transform［C］∥ European Conference on Computer Vision. Cham： Springer， 2016： 467-483. |
| 29 | WANG J， ZHOU F， WEN S L， et al. Deep metric learning with angular loss［C］∥ 2017 IEEE International Conference on Computer Vision （ICCV）. Piscataway： IEEE Press， 2017： 2612-2620. |
| 30 | HAN X F， LEUNG T， JIA Y Q， et al. MatchNet： Unifying feature and metric learning for patch-based matching［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 3279-3286. |
| 31 | ZAGORUYKO S， KOMODAKIS N. Learning to compare image patches via convolutional neural networks［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 4353-4361. |
| 32 | MA W P， ZHANG J， WU Y， et al. A novel two-step registration method for remote sensing images based on deep and local features［J］. IEEE Transactions on Geoscience and Remote Sensing， 2019， 57（7）： 4834-4843. |
| 33 | HUGHES L H， SCHMITT M， MOU L C， et al. Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN［J］. IEEE Geoscience and Remote Sensing Letters， 2018， 15（5）： 784-788. |
| 34 | HOFFMANN S， BRUST C A， SHADAYDEH M， et al. Registration of high resolution SAR and optical satellite imagery using fully convolutional networks［C］∥ IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2019： 5152-5155. |
| 35 | HUGHES L H， MARCOS D， LOBRY S， et al. A deep learning framework for matching of SAR and optical imagery［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2020， 169： 166-179. |
| 36 | QUAN D， WANG S， LIANG X F， et al. Deep generative matching network for optical and SAR image registration［C］∥ IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2018： 6215-6218. |
| 37 | MERKLE N， AUER S， MüLLER R， et al. Exploring the potential of conditional adversarial networks for optical and SAR image matching［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2018， 11（6）： 1811-1820. |
| 38 | BALNTAS V， RIBA E， PONSA D， et al. Learning local feature descriptors with triplets and shallow convolutional neural networks［C］∥ Proceedings of the British Machine Vision Conference 2016， 2016. |
| 39 | MISHCHUK A， MISHKIN D， RADENOVI? F， et al. Working hard to know your neighbor’s margins： Local descriptor learning loss［C］∥ Proceedings of the 31st International Conference on Neural Information Processing Systems. New York： ACM， 2017： 4829–4840. |
| 40 | TIAN Y R， FAN B， WU F C. L2-net： Deep learning of discriminative patch descriptor in euclidean space［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 6128-6136. |
| 41 | HE H Q， CHEN M， CHEN T， et al. Matching of remote sensing images with complex background variations via Siamese convolutional neural network［J］. Remote Sensing， 2018， 10（3）： 355. |
| 42 | HE H Q， CHEN M， CHEN T， et al. Learning to match multitemporal optical satellite images using multi-support-patches Siamese networks［J］. Remote Sensing Letters， 2019， 10（6）： 516-525. |
| 43 | LIAO Y， DI Y D， ZHOU H， et al. Feature matching and position matching between optical and SAR with local deep feature descriptor［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2021， 15： 448-462. |
| 44 | QUAN D， WEI H Y， WANG S， et al. Self-distillation feature learning network for optical and SAR image registration［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 4706718. |
| 45 | ZHU R J， YU D W， JI S P， et al. Matching RGB and infrared remote sensing images with densely-connected convolutional neural networks［J］. Remote Sensing， 2019， 11（23）： 2836. |
| 46 | MA J Y， JIANG X Y， FAN A X， et al. Image matching from handcrafted to deep features： A survey［J］. International Journal of Computer Vision， 2021， 129（1）： 23-79. |
| 47 | ZHANG H， LEI L， NI W P， et al. Explore better network framework for high-resolution optical and SAR image matching［J］. IEEE Transactions on Geoscience and Remote Sensing， 2021， 60： 4704418. |
| 48 | YE Y X， TANG T F， ZHU B， et al. A multiscale framework with unsupervised learning for remote sensing image registration［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5622215. |
| 49 | FANG Y Y， HU J， DU C， et al. SAR-optical image matching by integrating Siamese U-net with FFT correlation［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 19： 4016505. |
| 50 | LIN T Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］∥ 2017 IEEE International Conference on Computer Vision （ICCV）. Piscataway： IEEE Press， 2017： 2999-3007. |
| 51 | ZHENG L X， XIAO G B， SHI Z W， et al. MSA-net： Establishing reliable correspondences by multiscale attention network［J］. IEEE Transactions on Image Processing： a Publication of the IEEE Signal Processing Society， 2022， 31： 4598-4608. |
/
| 〈 |
|
〉 |
CJA