基于鱼眼相机和RGBD相机的异构协同SLAM

doi:10.7527/S1000-6893.2022.27621

Abstract

Abstract:

SLAM technology has a the potential of wide applications in autonomous navigation in satellite navigation denied environments. To combine the advantages of monocular fisheye SLAM to obtain more texture information and RGBD-SLAM to directly obtain scale information， a heterogeneous collaborative SLAM system is designed based on a monocular fisheye camera and a RGBD camera. Firstly， a method for checking the 3D gray centroid direction consistency between feature points is designed to screen the candidate matching points between heterogeneous images. Then， a step-by-step optical flow and projection matching method between heterogeneous images is designed to achieve high-performance feature point matching and relative pose estimation between fisheye and RGBD camera. Finally， based on the ORB-SLAM2 framework， a heterogeneous collaborative SLAM system is proposed based on fisheye and RGBD camera. The experimental results show that compared with traditional feature point matching methods， the proposed feature point matching method shows higher performance in the task of image feature matching with heterogeneous cameras. Compared with the monocular fisheye SLAM and RGBD-SLAM system， the proposed heterogeneous collaborative SLAM system has better performance under the conditions of rapid camera movement， camera close to the scene， low frame rate， texture loss， pure rotation of the camera， outdoor large scenes， etc.， and demonstrates improved robustness， anti-drift ability and trajectory accuracy.

Key words: heterogeneous collaborative SLAM, heterogeneous image feature matching, fisheye camera, RGBD camera, candidate matching points filtering

CLC Number:

V249.3

Yutong ZHANG, Jianmei SONG, Yan DING, Jinpeng LIU. Heterogeneous collaborative SLAM based on fisheye and RGBD cameras[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(10): 327621-327621.

Figures/Tables 29

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Fig. 17

Fig. 18

Table 1

Fig. 19

Fig. 20

Table 2

Fig. 21

Fig. 22

Fig. 23

Fig. 24

Fig. 25

Fig. 26

Fig. 27

References 39

1	赵良玉，金瑞，朱叶青，等. 基于点线特征融合的双目惯性SLAM算法［J］. 航空学报， 2022， 43（3）： 325117.
	ZHAO L Y， JIN R， ZHU Y Q， et al. Stereo visual-inertial slam with point and line features［J］. Acta Aeronautica et Astronautica Sinica， 2022， 43（3）： 325117 （in Chinese）.
2	谢洪乐，陈卫东，范亚娴，等. 月面特征稀疏环境下的视觉惯性SLAM方法［J］. 航空学报， 2021， 42（1）： 524169.
	XIE H L， CHEN W D， FAN Y X， et al. Visual-inertial SLAM in featureless environments on lunar surface［J］. Acta Aeronautica et Astronautica Sinica， 2021， 42（1）： 524169 （in Chinese）.
3	王小涛，张家友，王邢波，等. 基于FastSLAM的绳系机器人同时定位与地图构建算法［J］. 航空学报， 2021， 42（1）： 523893.
	WANG X T， ZHANG J Y， WANG X B， et al. Simultaneous localization and mapping algorithm based on FastSLAM framework for tethered robots［J］. Acta Aeronautica et Astronautica Sinica， 2021， 42（1）： 523893 （in Chinese）.
4	DAVISON A J， REID I D， MOLTON N D， et al. MonoSLAM： Real-time single camera SLAM［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2007， 29（6）： 1052-1067.
5	KLEIN G， MURRAY D. Parallel tracking and mapping for small AR workspaces［C］∥ 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. Piscataway： IEEE Press， 2008： 225-234.
6	KLEIN G， MURRAY D. Improving the agility of keyframe-based SLAM［C］∥European Conference on Computer Vision. Berlin： Springer， 2008： 802-815.
7	MUR-ARTAL R， MONTIEL J M M， TARDÓS J D. ORB-SLAM： A versatile and accurate monocular SLAM system［J］. IEEE Transactions on Robotics， 2015， 31（5）： 1147-1163.
8	LIN Y， GAO F， QIN T， et al. Autonomous aerial navigation using monocular visual-inertial fusion［J］. Journal of Field Robotics， 2018， 35（1）： 23-51.
9	WANG Y H， CAI S J， LI S J， et al. CubemapSLAM： A piecewise-pinhole monocular fisheye SLAM system［C］∥Asian Conference on Computer Vision. Cham： Springer， 2019： 34-49.
10	CARUSO D， ENGEL J， CREMERS D. Large-scale direct SLAM for omnidirectional cameras［C］∥2015 IEEE/RSJ International Conference on Intelligent Robots and Systems （IROS）. Piscataway： IEEE Press， 2015： 141-148.
11	ENGEL J， SCHÖPS T， CREMERS D. LSD-SLAM： Large-scale direct monocular SLAM［C］∥European Conference on Computer Vision. Cham： Springer， 2014： 834-849.
12	ZHANG Z C， REBECQ H， FORSTER C， et al. Benefit of large field-of-view cameras for visual odometry［C］∥ 2016 IEEE International Conference on Robotics and Automation （ICRA）. Piscataway： IEEE Press， 2016： 801-808.
13	FORSTER C， PIZZOLI M， SCARAMUZZA D. SVO： Fast semi-direct monocular visual odometry［C］∥2014 IEEE International Conference on Robotics and Automation （ICRA）. Piscataway： IEEE Press， 2014： 15-22.
14	MATSUKI H， VON STUMBERG L， USENKO V， et al. Omnidirectional DSO： Direct sparse odometry with fisheye cameras［J］. IEEE Robotics and Automation Letters， 2018， 3（4）： 3693-3700.
15	ENGEL J， KOLTUN V， CREMERS D. Direct sparse odometry［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（3）： 611-625.
16	JO Y G， HONG S H， HWANG S S， et al. Fisheye lens camera based autonomous valet parking system［DB/OL］. arXiv preprint： 2104.13119， 2021.
17	LIU S Y， GUO P， FENG L H， et al. Accurate and robust monocular SLAM with omnidirectional cameras［J］. Sensors （Basel， Switzerland）， 2019， 19（20）： 4494.
18	KHOMUTENKO B， GARCIA G， MARTINET P. An enhanced unified camera model［J］. IEEE Robotics and Automation Letters， 2016， 1（1）： 137-144.
19	CORREIA GARCIA T A， CAMPOS M B， CASTANHEIRO L F， et al. A proposal to integrate ORB-slam fisheye and convolutional neural networks for outdoor terrestrial mobile mapping［C］∥ 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. Piscataway： IEEE Press， 2021： 578-581.
20	URBAN S， HINZ S. MultiCol-SLAM - A modular real-time multi-camera SLAM system［DB/OL］. arXiv preprint： 1610.07336， 2016.
21	JI S P， QIN Z J， SHAN J. Panoramic SLAM from a multiple fisheye camera rig［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2020， 159： 169-183.
22	MUR-ARTAL R， TARDÓS J D. ORB-SLAM2： An open-source SLAM system for monocular， stereo， and RGB-D cameras［J］. IEEE Transactions on Robotics， 2017， 33（5）： 1255-1262.
23	ZHANG Y， HUANG F. Panoramic visual SLAM technology for spherical images［J］. Sensors （Basel， Switzerland）， 2021， 21（3）： 705.
24	ZHAO Q， FENG W， WAN L， et al. SPHORB： A fast and robust binary feature on the sphere［J］. International Journal of Computer Vision， 2015， 113（2）： 143-159.
25	AUDRAS C， COMPORT A I， MEILLAND M， et al. Real-time dense appearance-based SLAM for RGB-D sensors［C］∥Proceedings of the 2011 Australasian Conference on Robotics and Automation， 2011.
26	NEWCOMBE R A， IZADI S， HILLIGES O， et al. KinectFusion： Real-time dense surface mapping and tracking［C］∥ 2011 10th IEEE International Symposium on Mixed and Augmented Reality. Piscataway： IEEE Press， 2012： 127-136.
27	WHELAN T， JOHANNSSON H， KAESS M， et al. Robust real-time visual odometry for dense RGB-D mapping［C］∥ 2013 IEEE International Conference on Robotics and Automation. Piscataway： IEEE Press， 2013： 5724-5731.
28	CHEN J W， BAUTEMBACH D， IZADI S. Scalable real-time volumetric surface reconstruction［J］. ACM Transactions on Graphics， 2013， 32（4）： 1-16.
29	ENDRES F， HESS J， STURM J， et al. 3-D mapping with an RGB-D camera［J］. IEEE Transactions on Robotics， 2014， 30（1）： 177-187.
30	HENRY P， KRAININ M， HERBST E， et al. RGB-D mapping： Using depth cameras for dense 3D modeling of indoor environments［M］∥Experimental Robotics. Berlin： Springer， 2014： 477-491.
31	BESL P J， MCKAY N D. Method for registration of 3-D shapes ［J］. IEEE Transactions on Pattern Analysis & Machine Intelligence， 1992， 14（2）：239-256.
32	LOWE D G. Object recognition from local scale-invariant features［C］∥Proceedings of the Seventh IEEE International Conference on Computer Vision. Piscataway： IEEE Press， 2002： 1150-1157.
33	SUMIKURA S， SHIBUYA M， SAKURADA K. OpenVSLAM： A versatile visual SLAM framework［C］∥Proceedings of the 27th ACM International Conference on Multimedia. New York： ACM， 2019： 2292-2295.
34	CAMPOS C， ELVIRA R， RODRÍGUEZ J J G， et al. ORB-SLAM3： An accurate open-source library for visual， visual⁃inertial， and multimap SLAM［J］. IEEE Transactions on Robotics， 2021， 37（6）： 1874-1890.
35	YOUSIF K， TAGUCHI Y， RAMALINGAM S. MonoRGBD-SLAM： Simultaneous localization and mapping using both monocular and RGBD cameras［C］∥ 2017 IEEE International Conference on Robotics and Automation （ICRA）. Piscataway： IEEE Press， 2017： 4495-4502.
36	ZHANG Y T， SONG J M， DING Y， et al. FSD-BRIEF： A distorted BRIEF descriptor for fisheye image based on spherical perspective model［J］. Sensors （Basel， Switzerland）， 2021， 21（5）： 1839.
37	ROSTEN E， DRUMMOND T. Machine learning for high-speed corner detection［M］∥Computer Vision⁃ECCV 2006. Berlin： Springer， 2006： 430-443.
38	MIKOLAJCZYK K， SCHMID C. A performance evaluation of local descriptors［C］∥ 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Proceedings. Piscataway： IEEE Press， 2003： 1-7.
39	URBAN S， LEITLOFF J， HINZ S. MLPnP - A real-time maximum likelihood solution to the perspective-n-point problem［DB/OL］.arXiv preprint： 1607.08112， 2016.

数据集	匹配次数	本文算法		暴力匹配		BoW		FLANN
数据集	匹配次数	匹配成功次数	匹配成功率/%	匹配成功次数	匹配成功率/%	匹配成功次数	匹配成功率/%	匹配成功次数	匹配成功率/%
1-花园	1 309	1 252	95.6	590	45.1	0	0	147	11.2
2-走廊	1 535	1 529	99.6	1 122	73.1	574	37.4	401	26.1
3-实验室1	1 815	1 767	97.4	1 219	67.2	75	4.1	420	23.1
4-办公室1	1 609	1 483	92.2	753	46.8	58	3.6	352	21.9
5-实验室2	6 435	6 211	96.5	3 813	59.3	146	2.3	1 200	18.6
6-办公室2	1 831	1 771	96.7	1 010	55.2	184	10.0	559	30.5

[1]	. Multi-UAV decentralized cooperative navigation method based on memory-fusion [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 0, (): 0-0.
[2]	. Troposphere anomaly integrity monitoring parameters for GBAS [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 0, (): 0-0.
[3]	Huaijie ZHANG, Jingya MA, Haoyuan LIU, Pin GUO, Huichao DENG, Kun XU, Xilun DING. Indoor positioning technology of multi⁃rotor flying robot based on visual-inertial fusion [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(5): 426964-426964.
[4]	Jianyu SU, Haiyan FANG, Jingjing GAO, Liang ZHAO. Determination of optimal observation period for X-ray pulsar-based navigation [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(3): 526597-526597.
[5]	. PPP integrity monitoring algorithm for mass navigation applications [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 0, (): 0-0.
[6]	SUN Hongchi, MU Rongjun, LONG Teng, LI Shoupeng, CUI Naigang. Beidou/INS high dynamic deeply integrated navigation of near-space vehicle [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 325672-325672.
[7]	. L/C dual-band navigation signal modulation mode and performance evaluation of BeiDou system [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 0, (): 0-0.
[8]	ZHANG Zhouyu, CAO Yunfeng, FAN Yanming. Research progress of vision based aerospace conflict sensing technologies for small unmanned aerial vehicle in low altitude [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 25645-025645.
[9]	LI Ming, HU Hui, GUO Meng, YU Fei, XIE Hongqun. Unambiguous acquisition algorithm based on cross-correlation function shift multiplication for BOC (m,n) signals [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 325897-325897.
[10]	ZHAO Liangyu, JIN Rui, ZHU Yeqing, GAO Fengjie. Stereo visual-inertial SLAM algorithm based on merge of point and line features [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(3): 325117-325117.
[11]	LIU Shiming, LI Sihai, ZHENG Jiangtao, FU Qiangwen, TAO Yuanbo. Detection of slowly growing faults of GNSS/INS ultra-tight integration based on prefilters and two-stage AIME [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(3): 325168-325168.
[12]	HUANG Weiquan, FANG Tao, WANG Zongyi. Improved non-damping comprehensive calibration method for grid INS [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(9): 323921-323921.
[13]	NING Xiaolin, HUANG Yulin, CHAO Wen. Integrated navigation of solar disk velocity difference and sun direction for spacecraft [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(9): 324253-324253.
[14]	FAN Ying, HE Xiaofeng, FAN Chen, HU Xiaoping, WU Xuesong, HAN Guoliang, LUO Kaixin. Atmospheric polarized light orientation method in cloudy weather [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(9): 324263-324263.
[15]	NING Xiaolin, LIANG Xiaoyu, SUN Xiaohan, WANG Fan, WANG Longhua, FANG Jiancheng. Satellite stellar refraction navigation measurements and their performance: A comparison [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(8): 623536-623536.

Heterogeneous collaborative SLAM based on fisheye and RGBD cameras

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 29

References 39

Related Articles 15

Recommended Articles

Metrics

Comments