基于MatchNet和多点匹配约束的可见光-SAR图像匹配

doi:10.7527/S1000-6893.2023.29162

Abstract

Abstract:

Multimodal image matching between optical and SAR images is critical for visual navigation of the aircraft flying over complex areas. Due to different imaging mechanisms， accurate optical-SAR image matching faces several challenges， such as imaging geometry differences between optical and SAR， nonlinear radiation distortion of SAR， and noise interference. Although deep learning-based image matching methods have shown better adaptability than traditional methods， they still cannot fully solve the above-mentioned challenges and their matching accuracy may be not enough for visual navigation tasks. This paper proposes a new image matching method called Multi-point Matching Constraint MatchNet （MMC-MatchNet） based on the typical MatchNet framework， so as to improve the matching accuracy of the single template matching using MatchNet. The method consists of two main steps： sub-templates sampling and matching using MatchNet， and mismatched sub-templates removal based on multi-point geometric constraint. The original input image size of the MatchNet is maintained， while the matching accuracy for optical-SAR image matching is improved. This paper also proposes a multi-level training dataset generation method to train the MMC-MatchNet. Compared to the random sampling method， our method focuses on the difficult samples in image matching， and can be seen as a special data augmentation method for image matching. MMC-MatchNet is tested on two multi-modal image matching datasets， outperforming NCC， NMI， HOPC and the single-template MatchNet. With similar matching accuracy to that of HOPC， MMC-MatchNet can improve the Correctly Matching Rate （CMR）. The method based on multi-point matching constraint and multi-level dataset generation can be easily extend to other patch-based image matching models.

Key words: image matching, template matching, optical image, SAR image, MatchNet

CLC Number:

Yibin YE, Xichao TENG, Qifeng YU, Zhang LI. Optical⁃SAR image matching based on MatchNet and multi⁃point matching constraint[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(10): 329162-329162.

Figures/Tables 26

Fig.1

Table 1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Table 2

Comparison between MMC⁃MatchNet and other algorithms

方法	$C M R 10$ /%	$M E 10$
NMI	40.0	2.80
NCC	44.6	3.54
HOPC	83.3	1.82
$M a t c h N e t r e s i z e$	74.6	4.43
MMC-MatchNet	92.1	2.15

Table 2

Fig.10

Table 3

Table 4

Fig.11

Fig.12

Fig.13

Table 5

Influence of different removal methods of mismatched points on fine matching result

精匹配方法	$C M R 10$ /%	$M E 10$ /像素	计算时间/（ms·组^-1）
2点向量法	91.5	3.10	104
3点向量法	91.1	2.98	109
4点向量法	91.4	2.83	141
5点向量法	91.5	2.77	521
6点向量法	91.6	2.71	869
7点向量法	91.7	2.64	2 483
8点向量法	91.7	2.58	6 267
9点向量法	91.8	2.54	13 057
10点向量法	91.5	2.50	32 032
中值法	92.1	2.15	111
粗匹配结果	92.1	2.46

Table 5

Fig.14

Fig.15

Fig.16

Table 6

CMR10 of MMC⁃MatchNet on templates of different sizes

模板图尺寸	CMR₁₀/%
模板图尺寸	$124 × 124$	$172 × 172$	$256 × 256$
MMC-MatchNet	92.1	96.7	98.8
HOPC	83.3	91.3	98.5
NMI	40.6	55.7	76.7
NCC	44.6	48.2	57.4

Table 6

Table 7

Effect of Focal⁃Loss on matching results

损失函数	交叉熵损失	Focal-Loss
参数		$α t = 0.25, γ = 2$	$α t = 0.25, γ = 4$	$α t = 1, γ = 2$	$α t = 1, γ = 6$
验证集分类精度/%	95.8	95.5	94.7	95.6	94.5
测试集 $C M R 10$ /%	90.3	87.4	84.3	89.4	82.4

Table 7

Table 8

Fig. 17

Table 9

Effect of TD on matching results

$T D$	$r a t e f$ /%	$r a t e t$ /%
100	0	0
80	2.8	0.7
60	5.7	0.8
40	6.6	3.4
20	7.8	14.2

Table 9

References 51

1	刘中杰，曹云峰，庄丽葵，等. 基于控制线方法的机载SAR和可见光图像匹配应用研究［J］. 航空学报， 2013， 34（9）： 2194-2201.
	LIU Z J， CAO Y F， ZHUANG L K， et al. Applied research on airborne SAR and optical image registration based on control line method［J］. Acta Aeronautica et Astronautica Sinica， 2013， 34（9）： 2194-2201 （in Chinese）.
2	赵耀，熊智，田世伟，等. 基于SAR图像匹配结果可信度评价的INS/SAR自适应Kalman滤波算法［J］. 航空学报， 2019， 40（8）： 322850.
	ZHAO Y， XIONG Z， TIAN S W， et al. INS/SAR adaptive Kalman filtering algorithm based on credibility evaluation of SAR image matching results［J］. Acta Aeronautica et Astronautica Sinica， 2019， 40（8）： 322850 （in Chinese）.
3	LU J Z， HU M Q， DONG J， et al. A novel dense descriptor based on structure tensor voting for multi-modal image matching［J］. Chinese Journal of Aeronautics， 2020， 33（9）： 2408-2419.
4	JIANG X Y， MA J Y， XIAO G B， et al. A review of multimodal image matching： Methods and applications［J］. Information Fusion， 2021， 73（C）： 22-71.
5	LI Q， FU B H， DONG Y F. Registration of radar and optical satellite images using multiscale filter technique and information measure［M］∥ IMPERATORE P， RICCIO D， eds. Geoscience and Remote Sensing New Achievements. London： InTech， 2010.
6	HONG T D， SCHOWENGERDT R A. A robust technique for precise registration of radar and optical satellite images［J］. Photogrammetric Engineering & Remote Sensing， 2005， 71（5）： 585-593.
7	PAUL S， PATI U C. A comprehensive review on remote sensing image registration［J］. International Journal of Remote Sensing， 2021， 42（14）： 5400-5436.
8	ZHAO F， HUANG Q M， WEN G. Image matching by normalized cross-correlation［C］∥ 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings. Piscataway： IEEE Press， 2006： Ⅱ.
9	WELLS W M， VIOLA P， ATSUMI H， et al. Multi-modal volume registration by maximization of mutual information［J］. Medical Image Analysis， 1996， 1（1）： 35-51.
10	YE Y X， SHAN J， BRUZZONE L， et al. Robust registration of multimodal remote sensing images based on structural similarity［J］. IEEE Transactions on Geoscience and Remote Sensing， 2017， 55（5）： 2941-2958.
11	SURI S， REINARTZ P. Mutual-information-based registration of TerraSAR-X and ikonos imagery in urban areas［J］. IEEE Transactions on Geoscience and Remote Sensing， 2010， 48（2）： 939-949.
12	YE Y X， SHEN L， HAO M， et al. Robust optical-to-SAR image matching based on shape properties［J］. IEEE Geoscience and Remote Sensing Letters， 2017， 14（4）： 564-568.
13	FAN B， HUO C L， PAN C H， et al. Registration of optical and SAR satellite images by exploring the spatial relationship of the improved SIFT［J］. IEEE Geoscience and Remote Sensing Letters， 2013， 10（4）： 657-661.
14	HUANG L， LI Z. Feature-based image registration using the shape context［J］. International Journal of Remote Sensing， 2010， 31（8）： 2169-2177.
15	XIANG Y M， WANG F， YOU H J. OS-SIFT： A robust SIFT-like algorithm for high-resolution optical-to-SAR image registration in suburban areas［J］. IEEE Transactions on Geoscience and Remote Sensing， 2018， 56（6）： 3078-3090.
16	WANG Y G， YU X D， ZHANG Y， et al. An adaptive SAR and optical images registration approach based on SOI-SIFT［C］∥ IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2022： 2582-2585.
17	ZHANG X T， WANG Y H， LIU H W. Robust optical and SAR image registration based on OS-SIFT and cascaded sample consensus［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 19： 4011605.
18	FAN J W， YE Y X， LIU G C， et al. Phase congruency order-based local structural feature for SAR and optical image matching［J］. IEEE Geoscience and Remote Sensing Letters， 2022， 19： 4507105.
19	RAGURAM R， CHUM O， POLLEFEYS M， et al. USAC： A universal framework for random sample consensus［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（8）： 2022-2038.
20	ZHAO M， ZHANG G X， DING M. Heterogeneous self-supervised interest point matching for multi-modal remote sensing image registration［J］. International Journal of Remote Sensing， 2022， 43（3）： 915-931.
21	XIANG D L， XIE Y Z， CHENG J D， et al. Optical and SAR image registration based on feature decoupling network［J］. IEEE Transactions on Geoscience and Remote Sensing， 1858， 60： 5235913.
22	LIPMAN Y， YAGEV S， PORANNE R， et al. Feature matching with bounded distortion［J］. ACM Transactions on Graphics， 2014， 33（3）： 26.
23	MA J Y， ZHAO J， JIANG J J， et al. Locality preserving matching［J］. International Journal of Computer Vision， 2019， 127（5）： 512-531.
24	BIAN J W， LIN W Y， MATSUSHITA Y， et al. GMS： Grid-based motion statistics for fast， ultra-robust feature correspondence［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 2828-2837.
25	VERDIE Y， YI K M， FUA P， et al. TILDE： A temporally invariant learned DEtector［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 5279-5288.
26	ZHANG X， YU F X， KARAMAN S， et al. Learning discriminative and transformation covariant local feature detectors［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 4923-4931.
27	LUO Z X， SHEN T W， ZHOU L， et al. ContextDesc： Local descriptor augmentation with cross-modality context［C］∥ 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2019： 2522-2531.
28	YI K M， TRULLS E， LEPETIT V， et al. LIFT： Learned invariant feature transform［C］∥ European Conference on Computer Vision. Cham： Springer， 2016： 467-483.
29	WANG J， ZHOU F， WEN S L， et al. Deep metric learning with angular loss［C］∥ 2017 IEEE International Conference on Computer Vision （ICCV）. Piscataway： IEEE Press， 2017： 2612-2620.
30	HAN X F， LEUNG T， JIA Y Q， et al. MatchNet： Unifying feature and metric learning for patch-based matching［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 3279-3286.
31	ZAGORUYKO S， KOMODAKIS N. Learning to compare image patches via convolutional neural networks［C］∥ 2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2015： 4353-4361.
32	MA W P， ZHANG J， WU Y， et al. A novel two-step registration method for remote sensing images based on deep and local features［J］. IEEE Transactions on Geoscience and Remote Sensing， 2019， 57（7）： 4834-4843.
33	HUGHES L H， SCHMITT M， MOU L C， et al. Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN［J］. IEEE Geoscience and Remote Sensing Letters， 2018， 15（5）： 784-788.
34	HOFFMANN S， BRUST C A， SHADAYDEH M， et al. Registration of high resolution SAR and optical satellite imagery using fully convolutional networks［C］∥ IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2019： 5152-5155.
35	HUGHES L H， MARCOS D， LOBRY S， et al. A deep learning framework for matching of SAR and optical imagery［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2020， 169： 166-179.
36	QUAN D， WANG S， LIANG X F， et al. Deep generative matching network for optical and SAR image registration［C］∥ IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. Piscataway： IEEE Press， 2018： 6215-6218.
37	MERKLE N， AUER S， MÜLLER R， et al. Exploring the potential of conditional adversarial networks for optical and SAR image matching［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2018， 11（6）： 1811-1820.
38	BALNTAS V， RIBA E， PONSA D， et al. Learning local feature descriptors with triplets and shallow convolutional neural networks［C］∥ Proceedings of the British Machine Vision Conference 2016， 2016.
39	MISHCHUK A， MISHKIN D， RADENOVIĆ F， et al. Working hard to know your neighbor’s margins： Local descriptor learning loss［C］∥ Proceedings of the 31st International Conference on Neural Information Processing Systems. New York： ACM， 2017： 4829–4840.
40	TIAN Y R， FAN B， WU F C. L2-net： Deep learning of discriminative patch descriptor in euclidean space［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 6128-6136.
41	HE H Q， CHEN M， CHEN T， et al. Matching of remote sensing images with complex background variations via Siamese convolutional neural network［J］. Remote Sensing， 2018， 10（3）： 355.
42	HE H Q， CHEN M， CHEN T， et al. Learning to match multitemporal optical satellite images using multi-support-patches Siamese networks［J］. Remote Sensing Letters， 2019， 10（6）： 516-525.
43	LIAO Y， DI Y D， ZHOU H， et al. Feature matching and position matching between optical and SAR with local deep feature descriptor［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing， 2021， 15： 448-462.
44	QUAN D， WEI H Y， WANG S， et al. Self-distillation feature learning network for optical and SAR image registration［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 4706718.
45	ZHU R J， YU D W， JI S P， et al. Matching RGB and infrared remote sensing images with densely-connected convolutional neural networks［J］. Remote Sensing， 2019， 11（23）： 2836.
46	MA J Y， JIANG X Y， FAN A X， et al. Image matching from handcrafted to deep features： A survey［J］. International Journal of Computer Vision， 2021， 129（1）： 23-79.
47	ZHANG H， LEI L， NI W P， et al. Explore better network framework for high-resolution optical and SAR image matching［J］. IEEE Transactions on Geoscience and Remote Sensing， 2021， 60： 4704418.
48	YE Y X， TANG T F， ZHU B， et al. A multiscale framework with unsupervised learning for remote sensing image registration［J］. IEEE Transactions on Geoscience and Remote Sensing， 2022， 60： 5622215.
49	FANG Y Y， HU J， DU C， et al. SAR-optical image matching by integrating Siamese U-net with FFT correlation［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 19： 4016505.
50	LIN T Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］∥ 2017 IEEE International Conference on Computer Vision （ICCV）. Piscataway： IEEE Press， 2017： 2999-3007.
51	ZHENG L X， XIAO G B， SHI Z W， et al. MSA-net： Establishing reliable correspondences by multiscale attention network［J］. IEEE Transactions on Image Processing： a Publication of the IEEE Signal Processing Society， 2022， 31： 4598-4608.

模块名称	操作类型	输出维度	核尺寸	步长
Conv0	卷积	64×64×24	7×7	1
Pool0	池化	32×32×24	3×3	2
Conv1	卷积	32×32×64	5×5	1
Pool1	池化	16×16×64	3×3	2
Conv2	卷积	16×16×96	3×3	1
Pool3	卷积	16×16×96	3×3	1
Conv4	卷积	16×16×64	3×3	1
Pool4	池化	8×8×96	3×3	2
Bottleneck	全连接	可调整
FC1	全连接	可调整
FC2	全连接	可调整
FC3	全连接	2

编号	场景	地理坐标	SAR图像	可见光图像
1	城市	18°24'34″N 109°30'26″E	来源：TerraSAR 时间：2014/11/23 地面分辨率：5 m 尺寸：200×200	来源：Google Earth 时间：2014/12/31 地面分辨率：5 m 尺寸：200×200
2	河流	18°15'87″N 109°30'40″E
3	山地	18°39'43″N 109°26'49″E
4	港口	18°12'40″N 109°29'51″E
5	岛屿	18°18'55″N 109°25'31″E
6	机场	18°18'18″N 109°24'36″E
7	平原	18°25'21″N 109°26'30″E
8	戈壁1	18°24'34″N 109°30'26″E	来源：GF3 时间：2020/12/03 地面分辨率：5 m 尺寸：200×200	来源：Google Earth 时间：2018/12/31 地面分辨率：5 m 尺寸：200×200
9	戈壁2	18°15'87″N 109°30'40″E	来源：GF3 时间：2020/12/03 地面分辨率：5 m 尺寸：200×200	来源：Google Earth 时间：2018/12/31 地面分辨率：5 m 尺寸：200×200

方法		RANSAC	多点匹配约束方法
联系	算法目的	均是从已有的候选点对中剔除误匹配对，进而计算待匹配图像间的转换模型
联系	误差计算	当采用相同的转换模型时，2种方法的匹配误差计算公式相似
区别	应用背景	剔除特征匹配的误匹配对	剔除基于Patch的小模板匹配的误匹配对
	匹配候选点对	通过特征点提取和匹配得到候选点对，候选点数量及分布随机	通过分块匹配的方式得到候选点对，候选点数量及分布固定
	变换模型	单应性矩阵，仿射变换矩阵以及RST等	平移向量
	匹配内点生成方式	随机抽取m个点作为初始内点，基于变换模型迭代更新内点个数	内点数n固定，选择平移误差最小组合为内点
	参数设置	初始内点个数，误差阈值，最大迭代次数等	分块数量K和内点个数n

[1]	Haiqiao LIU, Meng LIU, Zichao GONG, Jing DONG. A review of image matching methods based on deep learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(3): 28796-028796.
[2]	Hongfa WAN, Shanshan LI, Chaozhen LAN, Mingzhi XIANG. UAV visual positioning method based on factor graph [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S1): 727627-727627.
[3]	SU Hang, XU Congan, YAO Libo, LI Jianwei, LING Qing, GAO Long. A lightweight oriented ship detection method in SAR images [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726922-726922.
[4]	LIU Fang, YANG Anzhe, WU Zhiwei. Adaptive Siamese network based UAV target tracking algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(1): 323423-323423.
[5]	ZHAO Yao, XIONG Zhi, TIAN Shiwei, LIU Jianye, CUI Yuchen. INS/SAR adaptive Kalman filtering algorithm based on credibility evaluation of SAR image matching results [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2019, 40(8): 322850-322850.
[6]	ZENG Qinghua, PAN Pengju, LIU Jianye, WANG Yunshu, LIU Sheng. Fast and accurate target positioning with large viewpoint based on inertial navigation system information [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2017, 38(8): 321171-321171.
[7]	WANG Cong, WANG Haipeng, XIONG Wei, HE You. Refined track initiation algorithm for partly resolvable group targets based on phase correlation [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2017, 38(4): 320299-320299.
[8]	TIAN Yugang, YANG Gui, WU Wei. A strict geometric calibration method for airborne hyperspectral sensors aided by high resolution images [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2015, 36(4): 1250-1258.
[9]	LIU Zhongjie, CAO Yunfeng, ZHUANG Likui, DING Meng. Applied Research on Airborne SAR and Optical Image Registration Based on Control Line Method [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(9): 2194-2201.
[10]	YU Helong, SU Hengqiang, WANG Yan, FENG Xue. Application of SUSAN Corner Detection and Matching Algorithm in High Temperature Deformation Measurement [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(5): 1064-1072.
[11]	LIU Shuaiqi, HU Shaohai, XIAO Yang. SAR Image De-noising Based on Complex Shearlet Transform Domain Gaussian Mixture Model [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(1): 173-180.
[12]	Ling Zhigang;Liang Yan;Pan Quan;Shen He;Cheng Yongmei. A Robust Multi-level Scene Matching Algorithm for Infrared and Visible Light Image [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2010, 31(6): 1185-1195.
[13]	HU Zheng-lei;SUN Jin-ping;YUAN Yun-neng;MAO Shi-yi. SAR Image Despeckling in Wavelet Domain Based on α-Stable Model [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2006, 27(5): 928-933.

Optical⁃SAR image matching based on MatchNet and multi⁃point matching constraint

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 26

References 51

Related Articles 13

Recommended Articles

Metrics

Comments