Acta Aeronautica et Astronautica Sinica ›› 2024, Vol. 45 ›› Issue (10): 329162-329162.doi: 10.7527/S1000-6893.2023.29162
• Electronics and Electrical Engineering and Control • Previous Articles Next Articles
Yibin YE, Xichao TENG, Qifeng YU, Zhang LI()
Received:
2023-06-13
Revised:
2023-07-17
Accepted:
2023-08-11
Online:
2024-05-25
Published:
2023-09-01
Contact:
Zhang LI
E-mail:zhangli_nudt@163.com
Supported by:
CLC Number:
Yibin YE, Xichao TENG, Qifeng YU, Zhang LI. Optical⁃SAR image matching based on MatchNet and multi⁃point matching constraint[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(10): 329162-329162.
Table 3
Description of self⁃built testing dataset
编号 | 场景 | 地理坐标 | SAR图像 | 可见光图像 |
---|---|---|---|---|
1 | 城市 | 18°24'34″N 109°30'26″E | 来源:TerraSAR 时间:2014/11/23 地面分辨率:5 m 尺寸:200×200 | 来源:Google Earth 时间:2014/12/31 地面分辨率:5 m 尺寸:200×200 |
2 | 河流 | 18°15'87″N 109°30'40″E | ||
3 | 山地 | 18°39'43″N 109°26'49″E | ||
4 | 港口 | 18°12'40″N 109°29'51″E | ||
5 | 岛屿 | 18°18'55″N 109°25'31″E | ||
6 | 机场 | 18°18'18″N 109°24'36″E | ||
7 | 平原 | 18°25'21″N 109°26'30″E | ||
8 | 戈壁1 | 18°24'34″N 109°30'26″E | 来源:GF3 时间:2020/12/03 地面分辨率:5 m 尺寸:200×200 | 来源:Google Earth 时间:2018/12/31 地面分辨率:5 m 尺寸:200×200 |
9 | 戈壁2 | 18°15'87″N 109°30'40″E |
Table 4
Differences between challenge match dataset and self⁃built dataset
差异点 | 挑战赛数据集 | 自建数据集 | 备注 |
---|---|---|---|
图像 来源 | SAR:TerraSAR 可见光:GF2 | SAR: TerraSAR和GF3 可见光: Google Earth | 不同卫星拍摄的图像在传感器参数、拍摄时间、角度等方面差异较大 |
地面 分辨率 | 3 m | 5 m | 在相同尺寸的图片中,自建数据集的地物尺寸更小, 覆盖范围更大 |
场景 | 以城市、平原为主, 含有部分河流、 机场、山地区域 | 城市、河流、山地、 港口、岛屿、机场、 平原、戈壁8种 场景 | 2数据集覆盖区域 无重叠,同时自建 数据集多出了港口、岛屿和戈壁场景 |
Table 5
Influence of different removal methods of mismatched points on fine matching result
精匹配方法 | 计算时间/(ms·组-1) | ||
---|---|---|---|
2点向量法 | 91.5 | 3.10 | 104 |
3点向量法 | 91.1 | 2.98 | 109 |
4点向量法 | 91.4 | 2.83 | 141 |
5点向量法 | 91.5 | 2.77 | 521 |
6点向量法 | 91.6 | 2.71 | 869 |
7点向量法 | 91.7 | 2.64 | 2 483 |
8点向量法 | 91.7 | 2.58 | 6 267 |
9点向量法 | 91.8 | 2.54 | 13 057 |
10点向量法 | 91.5 | 2.50 | 32 032 |
中值法 | 92.1 | 2.15 | 111 |
粗匹配结果 | 92.1 | 2.46 |
Table 8
Relation and difference between multi⁃point match constraint and RANSAC
方法 | RANSAC | 多点匹配约束方法 | |
---|---|---|---|
联系 | 算法目的 | 均是从已有的候选点对中剔除误匹配对,进而计算待匹配图像间的转换模型 | |
误差计算 | 当采用相同的转换模型时,2种方法的匹配误差计算公式相似 | ||
区别 | 应用背景 | 剔除特征匹配的误匹配对 | 剔除基于Patch的小模板匹配的误匹配对 |
匹配候选点对 | 通过特征点提取和匹配得到候选点对,候选点数量及分布随机 | 通过分块匹配的方式得到候选点对,候选点数量及分布固定 | |
变换模型 | 单应性矩阵,仿射变换矩阵以及RST等 | 平移向量 | |
匹配内点生成方式 | 随机抽取m个点作为初始内点,基于变换模型迭代更新内点个数 | 内点数n固定,选择平移误差最小组合为内点 | |
参数设置 | 初始内点个数,误差阈值,最大迭代次数等 | 分块数量K和内点个数n |
1 | 刘中杰, 曹云峰, 庄丽葵, 等. 基于控制线方法的机载SAR和可见光图像匹配应用研究[J]. 航空学报, 2013, 34(9): 2194-2201. |
LIU Z J, CAO Y F, ZHUANG L K, et al. Applied research on airborne SAR and optical image registration based on control line method[J]. Acta Aeronautica et Astronautica Sinica, 2013, 34(9): 2194-2201 (in Chinese). | |
2 | 赵耀, 熊智, 田世伟, 等. 基于SAR图像匹配结果可信度评价的INS/SAR自适应Kalman滤波算法[J]. 航空学报, 2019, 40(8): 322850. |
ZHAO Y, XIONG Z, TIAN S W, et al. INS/SAR adaptive Kalman filtering algorithm based on credibility evaluation of SAR image matching results[J]. Acta Aeronautica et Astronautica Sinica, 2019, 40(8): 322850 (in Chinese). | |
3 | LU J Z, HU M Q, DONG J, et al. A novel dense descriptor based on structure tensor voting for multi-modal image matching[J]. Chinese Journal of Aeronautics, 2020, 33(9): 2408-2419. |
4 | JIANG X Y, MA J Y, XIAO G B, et al. A review of multimodal image matching: Methods and applications[J]. Information Fusion, 2021, 73(C): 22-71. |
5 | LI Q, FU B H, DONG Y F. Registration of radar and optical satellite images using multiscale filter technique and information measure[M]∥ IMPERATORE P, RICCIO D, eds. Geoscience and Remote Sensing New Achievements. London: InTech, 2010. |
6 | HONG T D, SCHOWENGERDT R A. A robust technique for precise registration of radar and optical satellite images[J]. Photogrammetric Engineering & Remote Sensing, 2005, 71(5): 585-593. |
7 | PAUL S, PATI U C. A comprehensive review on remote sensing image registration[J]. International Journal of Remote Sensing, 2021, 42(14): 5400-5436. |
8 | ZHAO F, HUANG Q M, WEN G. Image matching by normalized cross-correlation[C]∥ 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings. Piscataway: IEEE Press, 2006: Ⅱ. |
9 | WELLS W M, VIOLA P, ATSUMI H, et al. Multi-modal volume registration by maximization of mutual information[J]. Medical Image Analysis, 1996, 1(1): 35-51. |
10 | YE Y X, SHAN J, BRUZZONE L, et al. Robust registration of multimodal remote sensing images based on structural similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5): 2941-2958. |
11 | SURI S, REINARTZ P. Mutual-information-based registration of TerraSAR-X and ikonos imagery in urban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(2): 939-949. |
12 | YE Y X, SHEN L, HAO M, et al. Robust optical-to-SAR image matching based on shape properties[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(4): 564-568. |
13 | FAN B, HUO C L, PAN C H, et al. Registration of optical and SAR satellite images by exploring the spatial relationship of the improved SIFT[J]. IEEE Geoscience and Remote Sensing Letters, 2013, 10(4): 657-661. |
14 | HUANG L, LI Z. Feature-based image registration using the shape context[J]. International Journal of Remote Sensing, 2010, 31(8): 2169-2177. |
15 | XIANG Y M, WANG F, YOU H J. OS-SIFT: A robust SIFT-like algorithm for high-resolution optical-to-SAR image registration in suburban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(6): 3078-3090. |
16 | WANG Y G, YU X D, ZHANG Y, et al. An adaptive SAR and optical images registration approach based on SOI-SIFT[C]∥ IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. Piscataway: IEEE Press, 2022: 2582-2585. |
17 | ZHANG X T, WANG Y H, LIU H W. Robust optical and SAR image registration based on OS-SIFT and cascaded sample consensus[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 19: 4011605. |
18 | FAN J W, YE Y X, LIU G C, et al. Phase congruency order-based local structural feature for SAR and optical image matching[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 4507105. |
19 | RAGURAM R, CHUM O, POLLEFEYS M, et al. USAC: A universal framework for random sample consensus[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 2022-2038. |
20 | ZHAO M, ZHANG G X, DING M. Heterogeneous self-supervised interest point matching for multi-modal remote sensing image registration[J]. International Journal of Remote Sensing, 2022, 43(3): 915-931. |
21 | XIANG D L, XIE Y Z, CHENG J D, et al. Optical and SAR image registration based on feature decoupling network[J]. IEEE Transactions on Geoscience and Remote Sensing, 1858, 60: 5235913. |
22 | LIPMAN Y, YAGEV S, PORANNE R, et al. Feature matching with bounded distortion[J]. ACM Transactions on Graphics, 2014, 33(3): 26. |
23 | MA J Y, ZHAO J, JIANG J J, et al. Locality preserving matching[J]. International Journal of Computer Vision, 2019, 127(5): 512-531. |
24 | BIAN J W, LIN W Y, MATSUSHITA Y, et al. GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence[C]∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2017: 2828-2837. |
25 | VERDIE Y, YI K M, FUA P, et al. TILDE: A temporally invariant learned DEtector[C]∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2015: 5279-5288. |
26 | ZHANG X, YU F X, KARAMAN S, et al. Learning discriminative and transformation covariant local feature detectors[C]∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2017: 4923-4931. |
27 | LUO Z X, SHEN T W, ZHOU L, et al. ContextDesc: Local descriptor augmentation with cross-modality context[C]∥ 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2019: 2522-2531. |
28 | YI K M, TRULLS E, LEPETIT V, et al. LIFT: Learned invariant feature transform[C]∥ European Conference on Computer Vision. Cham: Springer, 2016: 467-483. |
29 | WANG J, ZHOU F, WEN S L, et al. Deep metric learning with angular loss[C]∥ 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2017: 2612-2620. |
30 | HAN X F, LEUNG T, JIA Y Q, et al. MatchNet: Unifying feature and metric learning for patch-based matching[C]∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2015: 3279-3286. |
31 | ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks[C]∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2015: 4353-4361. |
32 | MA W P, ZHANG J, WU Y, et al. A novel two-step registration method for remote sensing images based on deep and local features[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7): 4834-4843. |
33 | HUGHES L H, SCHMITT M, MOU L C, et al. Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(5): 784-788. |
34 | HOFFMANN S, BRUST C A, SHADAYDEH M, et al. Registration of high resolution SAR and optical satellite imagery using fully convolutional networks[C]∥ IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. Piscataway: IEEE Press, 2019: 5152-5155. |
35 | HUGHES L H, MARCOS D, LOBRY S, et al. A deep learning framework for matching of SAR and optical imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 169: 166-179. |
36 | QUAN D, WANG S, LIANG X F, et al. Deep generative matching network for optical and SAR image registration[C]∥ IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. Piscataway: IEEE Press, 2018: 6215-6218. |
37 | MERKLE N, AUER S, MÜLLER R, et al. Exploring the potential of conditional adversarial networks for optical and SAR image matching[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(6): 1811-1820. |
38 | BALNTAS V, RIBA E, PONSA D, et al. Learning local feature descriptors with triplets and shallow convolutional neural networks[C]∥ Proceedings of the British Machine Vision Conference 2016, 2016. |
39 | MISHCHUK A, MISHKIN D, RADENOVIĆ F, et al. Working hard to know your neighbor’s margins: Local descriptor learning loss[C]∥ Proceedings of the 31st International Conference on Neural Information Processing Systems. New York: ACM, 2017: 4829–4840. |
40 | TIAN Y R, FAN B, WU F C. L2-net: Deep learning of discriminative patch descriptor in euclidean space[C]∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2017: 6128-6136. |
41 | HE H Q, CHEN M, CHEN T, et al. Matching of remote sensing images with complex background variations via Siamese convolutional neural network[J]. Remote Sensing, 2018, 10(3): 355. |
42 | HE H Q, CHEN M, CHEN T, et al. Learning to match multitemporal optical satellite images using multi-support-patches Siamese networks[J]. Remote Sensing Letters, 2019, 10(6): 516-525. |
43 | LIAO Y, DI Y D, ZHOU H, et al. Feature matching and position matching between optical and SAR with local deep feature descriptor[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 15: 448-462. |
44 | QUAN D, WEI H Y, WANG S, et al. Self-distillation feature learning network for optical and SAR image registration[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 4706718. |
45 | ZHU R J, YU D W, JI S P, et al. Matching RGB and infrared remote sensing images with densely-connected convolutional neural networks[J]. Remote Sensing, 2019, 11(23): 2836. |
46 | MA J Y, JIANG X Y, FAN A X, et al. Image matching from handcrafted to deep features: A survey[J]. International Journal of Computer Vision, 2021, 129(1): 23-79. |
47 | ZHANG H, LEI L, NI W P, et al. Explore better network framework for high-resolution optical and SAR image matching[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 4704418. |
48 | YE Y X, TANG T F, ZHU B, et al. A multiscale framework with unsupervised learning for remote sensing image registration[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5622215. |
49 | FANG Y Y, HU J, DU C, et al. SAR-optical image matching by integrating Siamese U-net with FFT correlation[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 19: 4016505. |
50 | LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]∥ 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE Press, 2017: 2999-3007. |
51 | ZHENG L X, XIAO G B, SHI Z W, et al. MSA-net: Establishing reliable correspondences by multiscale attention network[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2022, 31: 4598-4608. |
[1] | Haiqiao LIU, Meng LIU, Zichao GONG, Jing DONG. A review of image matching methods based on deep learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(3): 28796-028796. |
[2] | Hongfa WAN, Shanshan LI, Chaozhen LAN, Mingzhi XIANG. UAV visual positioning method based on factor graph [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S1): 727627-727627. |
[3] | SU Hang, XU Congan, YAO Libo, LI Jianwei, LING Qing, GAO Long. A lightweight oriented ship detection method in SAR images [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726922-726922. |
[4] | LIU Fang, YANG Anzhe, WU Zhiwei. Adaptive Siamese network based UAV target tracking algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(1): 323423-323423. |
[5] | ZHAO Yao, XIONG Zhi, TIAN Shiwei, LIU Jianye, CUI Yuchen. INS/SAR adaptive Kalman filtering algorithm based on credibility evaluation of SAR image matching results [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2019, 40(8): 322850-322850. |
[6] | ZENG Qinghua, PAN Pengju, LIU Jianye, WANG Yunshu, LIU Sheng. Fast and accurate target positioning with large viewpoint based on inertial navigation system information [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2017, 38(8): 321171-321171. |
[7] | WANG Cong, WANG Haipeng, XIONG Wei, HE You. Refined track initiation algorithm for partly resolvable group targets based on phase correlation [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2017, 38(4): 320299-320299. |
[8] | TIAN Yugang, YANG Gui, WU Wei. A strict geometric calibration method for airborne hyperspectral sensors aided by high resolution images [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2015, 36(4): 1250-1258. |
[9] | LIU Zhongjie, CAO Yunfeng, ZHUANG Likui, DING Meng. Applied Research on Airborne SAR and Optical Image Registration Based on Control Line Method [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(9): 2194-2201. |
[10] | YU Helong, SU Hengqiang, WANG Yan, FENG Xue. Application of SUSAN Corner Detection and Matching Algorithm in High Temperature Deformation Measurement [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(5): 1064-1072. |
[11] | LIU Shuaiqi, HU Shaohai, XIAO Yang. SAR Image De-noising Based on Complex Shearlet Transform Domain Gaussian Mixture Model [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(1): 173-180. |
[12] | Ling Zhigang;Liang Yan;Pan Quan;Shen He;Cheng Yongmei. A Robust Multi-level Scene Matching Algorithm for Infrared and Visible Light Image [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2010, 31(6): 1185-1195. |
[13] | HU Zheng-lei;SUN Jin-ping;YUAN Yun-neng;MAO Shi-yi. SAR Image Despeckling in Wavelet Domain Based on α-Stable Model [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2006, 27(5): 928-933. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||
Address: No.238, Baiyan Buiding, Beisihuan Zhonglu Road, Haidian District, Beijing, China
Postal code : 100083
E-mail:hkxb@buaa.edu.cn
Total visits: 6658907 Today visits: 1341All copyright © editorial office of Chinese Journal of Aeronautics
All copyright © editorial office of Chinese Journal of Aeronautics
Total visits: 6658907 Today visits: 1341