基于球形变换的无人机视频图像实时拼接方法

doi:10.7527/S1000-6893.2023.28364

Abstract

Abstract:

In the emergency situations such as fire situation monitoring， the image of the task area is obtained through UAV aerial photography. The traditional UAV image mosaic method has the problem of low timeliness. However， the real-time splicing based on video suffers from problem of homography error accumulation and splicing interruption. This paper designs and implements a real-time video image stitching method for UAV based on spherical transformation. After spherical transformation is added， global consistency can be increased for large angle stitching. In addition， only a single homography matrix is calculated， which ensures the splicing efficiency of the algorithm on the premise of angle robustness. This paper first calculates the SURF feature description of the generated image， uses the matching algorithm to match the Speeded Up Robust Features（SURF）features between consecutive images， and then uses the random sampling consistency algorithm to screen the matching. According to the matching relationship between images， the homography transformation matrix is calculated， and the error correction sphere parameters are calculated after the homography matrix is split. After the spherical transformation correction， homography is used to complete image mosaic， and finally image fusion is carried out. The experimental results show that the method proposed in this paper can improve the timeliness of traditional image mosaic（Compared with the Shape-Preserving Half-Projective （SPHP） algorithm， the tie splicing time has been reduced from 2 345.25 ms to 1 528.6 ms， resulting in a 34.8% improvement in algorithm efficiency） and the robustness of image splicing with the change of camera angle and it can still perform well in situations with large angles and poor visual perception.

Key words: UAV, image stitching, SURF, spherical transformation, homography transformation

CLC Number:

Guoqi ZENG, Zifan NIU, Lili ZHENG, Jie LI, Delin HAO. A real time video image stitching method for UAV based on spherical transformation[J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(24): 328364-328364.

Figures/Tables 23

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Table 1

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig. 14

Fig.15

Fig.16

Fig.17

Fig.18

Fig.19

Fig.20

Table 2

Table 3

References 20

1	MORANDO L， RECCHIUTO C T， CALLA J， et al. Thermal and visual tracking of photovoltaic plants for autonomous UAV inspection［J］. Drones， 2022， 6（11）： 347.
2	ZHANG Y T， CHEN S， WANG W Y， et al. Pyramid attention based early forest fire detection using UAV imagery［J］. Journal of Physics： Conference Series， 2022， 2363（1）： 012021.
3	CARVAJAL-RAMÍREZ F， SERRANO J M P R， AGÜERA-VEGA F， et al. A comparative analysis of phytovolume estimation methods based on UAV-photogrammetry and multispectral imagery in a Mediterranean forest［J］. Remote Sensing， 2019， 11（21）： 2579.
4	SKONDRAS A， KARACHALIOU E， TAVANTZIS I， et al. UAV mapping and 3D modeling as a tool for promotion and management of the urban space［J］. Drones， 2022， 6（5）： 115.
5	SHAO R Z， DU C， CHEN H， et al. Fast anchor point matching for emergency UAV image stitching using position and pose information［J］. Sensors， 2020， 20（7）： 2007.
6	ABBADI N K E， HASSANI S A AL， ABDULKHALEQ A H. A review over panoramic image stitching techniques［J］. Journal of Physics： Conference Series， 2021， 1999（1）： 012115.
7	LOWE D G. Object recognition from local scale-invariant features［C］∥ Proceedings of the Seventh IEEE International Conference on Computer Vision. Piscataway： IEEE Press， 2002： 1150-1157.
8	BAY H， ESS A， TUYTELAARS T， et al. Speeded-up robust features （SURF）［J］. Computer Vision and Image Understanding， 2008， 110（3）： 346-359.
9	FISCHER P， DOSOVITSKIY A， BROX T. Descriptor matching with convolutional neural networks： A comparison to SIFT［J］. Computer Science， 2014， 4（7）： 678-694.
10	NGUYEN T， CHEN S W， SHIVAKUMAR S S， et al. Unsupervised deep homography： A fast and robust homography estimation model［J］. IEEE Robotics and Automation Letters， 2018， 3（3）： 2346-2353.
11	ZHANG Y J， MEI X G， MA Y， et al. Hyperspectral panoramic image stitching using robust matching and adaptive bundle adjustment［J］. Remote Sensing， 2022， 14（16）： 4038.
12	ZARAGOZA J， CHIN T J， BROWN M S， et al. As-projective-as-possible image stitching with moving DLT［C］∥ 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2013： 2339-2346.
13	CHANG C H， SATO Y， CHUANG Y Y. Shape-preserving half-projective warps for image stitching［C］∥ 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2014： 3254-3261.
14	CHEN Y S， CHUANG Y Y. Natural image stitching with the global similarity prior［M］∥ Computer Vision⁃ECCV 2016. Cham： Springer International Publishing， 2016： 186-201.
15	ZHANG G F， HE Y， CHEN W F， et al. Multi-viewpoint panorama construction with wide-baseline images［C］∥ IEEE Transactions on Image Processing. Piscataway： IEEE Press， 2016： 3099-3111.
16	LIN K M， JIANG N J， LIU S C， et al. Direct photometric alignment by mesh deformation［C］∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2017： 2701-2709.
17	LEE K Y， SIM J Y. Warping residual based image stitching for large parallax［C］∥ 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Piscataway： IEEE Press， 2020： 8195-8203.
18	DE A LIMA NETO E， RODRIGUES P C. Kernel robust singular value decomposition［J］. Expert Systems With Applications， 2023， 211： 118555.
19	LI X， LIU Y R， LI D H， et al. Spherical image stitching based on ORB and PROSAC algorithm［C］∥ Proceedings of the 3rd International Conference on Intelligent Information Processing. New York： ACM， 2018： 160-165.
20	KIM T， IM Y J. Automatic satellite image registration by combination of matching and random sample consensus［J］. IEEE Transactions on Geoscience and Remote Sensing， 2003， 41（5）： 1111-1117.

算法	特征/个	匹配点/个	运行时间/ms	千点运行时间/ms
ORB	500	161	345.2	690.6
SURF	796	398	1 726.48	2 168

算法	耗时/ms	平均耗时/ms	输出大小/KB
SPHP	351 788	2 345.25	3 915
ODM	858 904	5 726.48	4 330
Ball	285 900	1 906.11	3 860

算法	耗时/ms	平均耗时/ms	输出大小/KB
SPHP	351 788	2 345.25	3 915
ODM	858 904	5 726.48	4 330
Ball	229 290	1 528.60	3 817

[1]	Haifeng WANG, Kunpeng LIU, Hongxin JIANG, Chenxi DU. Aerodynamic optimization method of propeller multi⁃design points and variable pitch angle strategy [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(9): 528831-528831.
[2]	Jing ZHAO, Dan SONG. Integrity monitoring method for GNSS/IMU integrated navigation system of UAV [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(7): 328943-328943.
[3]	Chuanyun WANG, Yang SU, Linlin WANG, Tian WANG, Jingjing WANG, Qian GAO. Multi-object continuous robust tracking algorithm for anti-UAV swarm [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(7): 329017-329017.
[4]	Yanhua ZHANG, Dengcheng ZHANG, Zhangwen ZHOU, Yuchang LEI, Lin LI. Concept and design of virtual rudder surface aircraft based on circulation control: Review [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 629608-629608.
[5]	Hongyu YIN, Yu WU, Tianjiao LIANG. Cooperative path planning for patrol coverage of fixed wing UAV [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 328944-328944.
[6]	Xudong LUO, Yiquan WU, Jinlin CHEN. Research progress on deep learning methods for object detection and semantic segmentation in UAV aerial images [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 28822-028822.
[7]	Chuihuan KONG, Dawei WU, Zhaoguang TAN, Lijun PAN, Rubing MA, Jiangtao SI. Design of fully electric scheme for three⁃surface verification aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 629618-629618.
[8]	Guangjia LI, Hongbo WANG, Kai ZHANG, Zhisheng YI. Lift enhancement and drag reduction technologies of solar powered unmanned aerial vehicles in near space: Review [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529644-529644.
[9]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.
[10]	Zhu WANG, Mengtong ZHANG, Zhenpeng ZHANG, Guangtong XU. Multi-UAV cooperative path planning based on multi-index dynamic priority [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328816-328816.
[11]	Gaojie ZHENG, Xiaoming HE, Dongpo LI, Huijun TAN, Kun WANG, Zhenlong WU, Depeng WANG. Double 90° deflection inlet/volute coupling flow characteristics of tail-powered unmanned aerial vehicle [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 128782-128782.
[12]	Chuankai LIU, Zhaoxiang WANG, Junxiong LEI, Zuoyu ZHANG, Kuangang FAN, Jitao ZHANG, Xiaoxue WANG, Hailang PAN, Jianguo LIU. An epipolar relaxation constrained matching algorithm of large-affined images for lunar rover with large span distance in a single movement [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(2): 328659-328659.
[13]	Hong WEI, Peng WANG, Wei DONG, Xiaofeng GUO, Zhida LI, Xuesen YANG, Zhongfu TANG, Chao FU. Effects of main flow velocity on frosting and defrosting characteristics of microtubule precooler [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(2): 128639-128639.
[14]	Xin HE, Zongying SHI, Yisheng ZHONG. Multi⁃USV cooperative collision avoidance based on velocity obstacle [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729758-729758.
[15]	Hongzhen GUO, Mou CHEN, Yongdong DAI, Maofei WANG. Distributed adaptive event⁃triggered formation control for QUAVs [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729917-729917.

A real time video image stitching method for UAV based on spherical transformation

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 23

References 20

Related Articles 15

Recommended Articles

Metrics

Comments