基于球形变换的无人机视频图像实时拼接方法
收稿日期: 2022-12-06
修回日期: 2023-02-06
录用日期: 2023-03-20
网络出版日期: 2023-05-06
A real time video image stitching method for UAV based on spherical transformation
Received date: 2022-12-06
Revised date: 2023-02-06
Accepted date: 2023-03-20
Online published: 2023-05-06
针对火灾态势监测等应急场景,需要通过无人机航拍获得现场的态势图,传统的基于无人机拍摄图片拼接方法通常存在时效性低的问题,而基于视频实时拼接方法,存在单应性误差积累,拼接中断的问题。设计并实现了一种基于球形变换的无人机实时视频图像拼接方法,加入球形变换后可以为大角度拼接增加全局一致性,并且仅仅计算单个单应性矩阵,保证了算法在角度鲁棒性下的拼接效率。首先,计算生成图像的加速鲁棒特征(SURF)特征描述,用匹配算法对连续图像之间的SURF特征进行匹配,然后,用随机抽样一致性算法对匹配进行筛选,根据图像之间的匹配关系,计算单应性变换矩阵,拆分单应性矩阵后计算误差校正球参数,在经过球形变换校正后利用单应性完成图像拼接,最后,进行图像融合。试验结果表明:所提的基于球形变换的视频图像拼接方法可以提高传统图像拼接的时效性(与形状保留半投影(SPHP)算法相比,平均每2张拼接时间从2 345.25 ms减少到1 528.6 ms,算法效率提升了34.8%)和相机角度变化下的鲁棒性,且在大角度、大视觉差的情况下仍能有着良好的表现。
曾国奇 , 牛子凡 , 郑丽丽 , 李杰 , 郝得霖 . 基于球形变换的无人机视频图像实时拼接方法[J]. 航空学报, 2023 , 44(24) : 328364 -328364 . DOI: 10.7527/S1000-6893.2023.28364
In the emergency situations such as fire situation monitoring, the image of the task area is obtained through UAV aerial photography. The traditional UAV image mosaic method has the problem of low timeliness. However, the real-time splicing based on video suffers from problem of homography error accumulation and splicing interruption. This paper designs and implements a real-time video image stitching method for UAV based on spherical transformation. After spherical transformation is added, global consistency can be increased for large angle stitching. In addition, only a single homography matrix is calculated, which ensures the splicing efficiency of the algorithm on the premise of angle robustness. This paper first calculates the SURF feature description of the generated image, uses the matching algorithm to match the Speeded Up Robust Features(SURF)features between consecutive images, and then uses the random sampling consistency algorithm to screen the matching. According to the matching relationship between images, the homography transformation matrix is calculated, and the error correction sphere parameters are calculated after the homography matrix is split. After the spherical transformation correction, homography is used to complete image mosaic, and finally image fusion is carried out. The experimental results show that the method proposed in this paper can improve the timeliness of traditional image mosaic(Compared with the Shape-Preserving Half-Projective (SPHP) algorithm, the tie splicing time has been reduced from 2 345.25 ms to 1 528.6 ms, resulting in a 34.8% improvement in algorithm efficiency) and the robustness of image splicing with the change of camera angle and it can still perform well in situations with large angles and poor visual perception.
Key words: UAV; image stitching; SURF; spherical transformation; homography transformation
1 | MORANDO L, RECCHIUTO C T, CALLA J, et al. Thermal and visual tracking of photovoltaic plants for autonomous UAV inspection[J]. Drones, 2022, 6(11): 347. |
2 | ZHANG Y T, CHEN S, WANG W Y, et al. Pyramid attention based early forest fire detection using UAV imagery[J]. Journal of Physics: Conference Series, 2022, 2363(1): 012021. |
3 | CARVAJAL-RAMíREZ F, SERRANO J M P R, AGüERA-VEGA F, et al. A comparative analysis of phytovolume estimation methods based on UAV-photogrammetry and multispectral imagery in a Mediterranean forest[J]. Remote Sensing, 2019, 11(21): 2579. |
4 | SKONDRAS A, KARACHALIOU E, TAVANTZIS I, et al. UAV mapping and 3D modeling as a tool for promotion and management of the urban space[J]. Drones, 2022, 6(5): 115. |
5 | SHAO R Z, DU C, CHEN H, et al. Fast anchor point matching for emergency UAV image stitching using position and pose information[J]. Sensors, 2020, 20(7): 2007. |
6 | ABBADI N K E, HASSANI S A AL, ABDULKHALEQ A H. A review over panoramic image stitching techniques[J]. Journal of Physics: Conference Series, 2021, 1999(1): 012115. |
7 | LOWE D G. Object recognition from local scale-invariant features[C]∥ Proceedings of the Seventh IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2002: 1150-1157. |
8 | BAY H, ESS A, TUYTELAARS T, et al. Speeded-up robust features (SURF)[J]. Computer Vision and Image Understanding, 2008, 110(3): 346-359. |
9 | FISCHER P, DOSOVITSKIY A, BROX T. Descriptor matching with convolutional neural networks: A comparison to SIFT[J]. Computer Science, 2014, 4(7): 678-694. |
10 | NGUYEN T, CHEN S W, SHIVAKUMAR S S, et al. Unsupervised deep homography: A fast and robust homography estimation model[J]. IEEE Robotics and Automation Letters, 2018, 3(3): 2346-2353. |
11 | ZHANG Y J, MEI X G, MA Y, et al. Hyperspectral panoramic image stitching using robust matching and adaptive bundle adjustment[J]. Remote Sensing, 2022, 14(16): 4038. |
12 | ZARAGOZA J, CHIN T J, BROWN M S, et al. As-projective-as-possible image stitching with moving DLT[C]∥ 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2013: 2339-2346. |
13 | CHANG C H, SATO Y, CHUANG Y Y. Shape-preserving half-projective warps for image stitching[C]∥ 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2014: 3254-3261. |
14 | CHEN Y S, CHUANG Y Y. Natural image stitching with the global similarity prior[M]∥ Computer Vision?ECCV 2016. Cham: Springer International Publishing, 2016: 186-201. |
15 | ZHANG G F, HE Y, CHEN W F, et al. Multi-viewpoint panorama construction with wide-baseline images[C]∥ IEEE Transactions on Image Processing. Piscataway: IEEE Press, 2016: 3099-3111. |
16 | LIN K M, JIANG N J, LIU S C, et al. Direct photometric alignment by mesh deformation[C]∥ 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2017: 2701-2709. |
17 | LEE K Y, SIM J Y. Warping residual based image stitching for large parallax[C]∥ 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2020: 8195-8203. |
18 | DE A LIMA NETO E, RODRIGUES P C. Kernel robust singular value decomposition[J]. Expert Systems With Applications, 2023, 211: 118555. |
19 | LI X, LIU Y R, LI D H, et al. Spherical image stitching based on ORB and PROSAC algorithm[C]∥ Proceedings of the 3rd International Conference on Intelligent Information Processing. New York: ACM, 2018: 160-165. |
20 | KIM T, IM Y J. Automatic satellite image registration by combination of matching and random sample consensus[J]. IEEE Transactions on Geoscience and Remote Sensing, 2003, 41(5): 1111-1117. |
/
〈 |
|
〉 |