基于视觉图像的空对空多无人机目标跟踪

doi:10.7527/S1000-6893.2023.29379

Abstract

Abstract:

Vision-based air-to-air multi-object tracking is a key technology for UAV situational awareness. Recent research is limited to single-UAV tracking and migration of general multi-object tracking algorithms. To address the problem of inaccurate air-to-air multi-UAVs tracking， a cascade multi-UAVs tracking algorithm based on block feature enhancement extraction and local geometric association is designed. The UAV image is processed in blocks according to the characteristics of the fuselage and the arm， and the UAV’s fine-grained morphological features are extracted. The property that the relative geometric relationship between the UAVs is virtually invariant in the continuous frames is also utilized to extract the local geometric vectors of the UAVs. Then， the cascade association algorithm is designed by synthesizing the above technical components to improve the multi-object tracking algorithm’s ability to retrieve UAV objects and the association success rate， so as to improve the tracking performance of the proposed algorithm. Experiments show that in the test set， the proposed algorithm improves ID F1 Score （IDF1） by 5.6% compared to the state-of-the-art multi-object tracking algorithms OC-SORT， and improves Multiple Object Tracking Accuracy （MOTA） by 2.7% compared to the ByteTrack， which also performs well in the general multi-object tracking. The optimal performance for air-to-air multi-UAVs tracking can be realized， and the components used by the proposed algorithm can also be applied to SORT， BYTE and other data association algorithms to jointly improve their performance.

Key words: Unmanned Aerial Vehicle (UAV), multi-object tracking, deep learning, feature extraction, data association

CLC Number:

V279

Zhaochen CHU, Tao SONG, Ren JIN, Defu LIN. Vision-based air-to-air multi-UAVs tracking[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(14): 629379.

Figures/Tables 17

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig. 6

Fig.7

Table 1

Table 2

Table 3

Performance of feature extraction component with different splitting ratios along with direction under different datasets

特征提取组件	mAP/%
特征提取组件	MOT-FLY	DUT Anti-UAV Tracking
ResNet50-BE（ $p = 1 ∶ 6 ∶ 1$ ）	64.9	61.9
ResNet50-BE（ $p = 1 ∶ 2 ∶ 1$ ）	69.6	66.8
ResNet50-BE（ $p = 3 ∶ 2 ∶ 3$ ）	68.4	64.0

Table 3

Table 4

Table 5

Table 6

Fig.8

Fig.9

Table 7

Table 8

References 39

1	陈琳，刘允刚. 面向无人机的视觉目标跟踪算法：综述与展望［J］. 信息与控制， 2022， 51（1）： 23-40.
	CHEN L， LIU Y G. UAV visual target tracking algorithms： Review and future prospect［J］. Information and Control， 2022， 51（1）： 23-40 （in Chinese）.
2	闫超，涂良辉，王聿豪，等. 无人机在我国民用领域应用综述［J］. 飞行力学， 2022， 40（3）： 1-6.
	YAN C， TU L H， WANG Y H， et al. Application of unmanned aerial vehicle in civil field in China［J］. Flight Dynamics， 2022， 40（3）： 1-6 （in Chinese）.
3	李仕帆. 无人机在消防灭火救援工作中的运用实践探讨［J］. 中国设备工程， 2023（8）： 8-10.
	LI S F. Practical discussion on the utilization of UAVs in fire fighting and rescue work［J］. China Plant Engineering， 2023（8）： 8-10 （in Chinese）.
4	李浩，孙合敏，李宏权，等. 无人机集群蜂群作战综述及其预警探测应对策略［J］. 飞航导弹， 2018（11）： 46-51.
	LI H， SUN H M， LI H Q， et al. Overview of UAV cluster swarm combat and its early warning detection response strategy［J］. Aerodynamic Missile Journal， 2018（11）： 46-51 （in Chinese）.
5	刘雷，刘大卫，王晓光，等. 无人机集群与反无人机集群发展现状及展望［J］. 航空学报， 2022， 43（S1）： 726908.
	LIU L， LIU D W， WANG X G，et al. Development status and outlook of UAV clusters and anti-UAV clusters［J］. Acta Aeronautica et Astronautica Sinica， 2022， 43（S1）： 726908 （in Chinese）.
6	WANG C Y， SU Y， WANG J J， et al. UAVSwarm dataset： An unmanned aerial vehicle swarm dataset for multiple object tracking［J］. Remote Sensing， 2022， 14（11）： 2601.
7	张洲宇，曹云峰，范彦铭. 低空小型无人机空域冲突视觉感知技术研究进展［J］. 航空学报， 2022， 43（8）： 025645.
	ZHANG Z Y， CAO Y F， FAN Y M. Research progress of vision based aerospace conflict sensing technologies for small unmanned aerial vehicle in low altitude［J］. Acta Aeronautica et Astronautica Sinica， 2022， 43（8）： 025645 （in Chinese）.
8	薛远亮，金国栋，谭力宁，等. 基于多尺度融合的自适应无人机目标跟踪算法［J］. 航空学报， 2023， 44（1）： 326107.
	XUE Y L， JIN G D， TAN L N， et al. Adaptive UAV target tracking algorithm based on multi-scale fusion［J］. Acta Aeronautica et Astronautica Sinica， 2023， 44（1）： 326107 （in Chinese）.
9	刘芳，孙亚楠. 基于自适应融合网络的无人机目标跟踪算法［J］. 航空学报， 2022， 43（7）： 325522.
	LIU F， SUN Y N. UAV target tracking algorithm based on adaptive fusion network［J］. Acta Aeronautica et Astronautica Sinica， 2022， 43（7）： 325522 （in Chinese）.
10	刘贞报，马博迪，高红岗，等. 基于形态自适应网络的无人机目标跟踪方法［J］. 航空学报， 2021， 42（4）： 524904.
	LIU Z B， MA B D， GAO H G， et al. Adaptive morphological network based UAV target tracking algorithm［J］. Acta Aeronautica et Astronautica Sinica， 2021， 42（4）： 524904 （in Chinese）.
11	DU Y H， WAN J F， ZHAO Y Y， et al. GIAOTracker： A comprehensive framework for MCMOT with global information and optimizing strategies in VisDrone 2021［C］∥2021 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway： IEEE Press， 2021： 2809-2819.
12	SHEN H Y， HUANG Z Y， WANG H C， et al. Adaptive update of UAV multi-target tracking based on Transformer［C］∥2023 IEEE 6th Information Technology， Networking， Electronic and Automation Control Conference. Piscataway： IEEE Press， 2023： 6-13.
13	WANG G A， GU R S， LIU Z Z， et al. Track without appearance： Learn box and tracklet embedding with local and global motion patterns for vehicle tracking［C］∥2021 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE Press， 2021： 9856-9866.
14	WU H， NIE J H， HE Z W， et al. One-shot multiple object tracking in UAV videos using task-specific fine-grained features［J］. Remote Sensing， 2022， 14（16）： 3853.
15	罗茜，赵睿，庄慧珊，等. YOLOv5与Deep-SORT联合优化的无人机多目标跟踪算法［J］. 信号处理， 2022， 38（12）： 2628-2638.
	LUO X， ZHAO R， ZHUANG H S， et al. UAV multi-target tracking algorithm jointly optimized by YOLOv5 and Deep-SORT［J］. Journal of Signal Processing， 2022， 38（12）： 2628-2638 （in Chinese）.
16	刘芳，浦昭辉，张帅超. 基于注意力特征融合的无人机多目标跟踪算法［J］. 控制与决策， 2023， 38（2）： 345-353.
	LIU F， PU Z H， ZHANG S C. UAV multi-target tracking algorithm based on attention feature fusion［J］. Control and Decision， 2023， 38（2）： 345-353 （in Chinese）.
17	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］∥Proceedings of the 31st International Conference on Neural Information Processing Systems. New York： ACM， 2017： 6000-6010.
18	ZHANG Y F， WANG C Y， WANG X G， et al. FairMOT： On the fairness of detection and re-identification in multiple object tracking［J］. International Journal of Computer Vision， 2021， 129（11）： 3069-3087.
19	ZHAO J， ZHANG J S， LI D D， et al. Vision-based anti-UAV detection and tracking［J］. IEEE Transactions on Intelligent Transportation Systems， 2022， 23（12）： 25323-25334.
20	CAO Z A， HUANG Z Y， PAN L， et al. TCTrack： Temporal contexts for aerial tracking［C］∥2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2022： 14778-14788.
21	LI Y M， FU C H， DING F Q， et al. AutoTrack： Towards high-performance visual tracking for UAV with automatic spatio-temporal regularization［C］∥2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2020： 11920-11929.
22	JIANG X L， LI P Z， LI Y J， et al. Graph neural based end-to-end data association framework for online multiple-object tracking［DB/OL］. arXiv preprint： 1907.05315， 2019.
23	ZHANG Y F， SUN P Z， JIANG Y， et al. ByteTrack： Multi-object tracking by associating every detection box［C］∥European Conference on Computer Vision. Cham： Springer， 2022： 1-21.
24	LUO W H， XING J L， MILAN A， et al. Multiple object tracking： A literature review［J］. Artificial Intelligence， 2021， 293： 103448.
25	WOJKE N， BEWLEY A， PAULUS D. Simple online and realtime tracking with a deep association metric［C］∥2017 IEEE International Conference on Image Processing. Piscataway： IEEE Press， 2017： 3645-3649.
26	DOSOVITSKIY A， BEYER L， KOLESNIKOV A， et al. An image is worth 16x16 words： Transformers for image recognition at scale［DB/OL］. arXiv preprint： 2010.11929， 2020.
27	REN S Q， HE K M， GIRSHICK R， et al. Faster R-CNN： Towards real-time object detection with region proposal networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（6）： 1137-1149.
28	BERGMANN P， MEINHARDT T， LEAL-TAIXÉ L. Tracking without bells and whistles［C］∥2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE Press， 2019： 941-951.
29	BEWLEY A， GE Z Y， OTT L， et al. Simple online and realtime tracking［C］∥2016 IEEE International Conference on Image Processing. Piscataway： IEEE Press， 2016： 3464-3468.
30	CAO J K， PANG J M， WENG X S， et al. Observation-centric SORT： Rethinking SORT for robust multi-object tracking［C］∥2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2023： 9686-9696.
31	ZHOU X Y， KOLTUN V， KRÄHENBÜHL P. Tracking objects as points［C］∥European Conference on Computer Vision. Cham： Springer， 2020： 474-490.
32	PENG J L， WANG C G， WAN F B， et al. Chained-tracker： Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking［C］∥European Conference on Computer Vision. Cham： Springer， 2020： 145-161.
33	GE Z， LIU S T， WANG F， et al. YOLOX： Exceeding YOLO series in 2021［DB/OL］. arXiv preprint： 2107.08430， 2021.
34	BOCHKOVSKIY A， WANG C Y， LIAO H Y M. YOLOv4： Optimal speed and accuracy of object detection［DB/OL］. arXiv preprint： 2004.10934， 2020.
35	ZHANG H Y， CISSE M， DAUPHIN Y N， et al. Mixup： Beyond empirical risk minimization［DB/OL］. arXiv preprint： 1710.09412， 2017.
36	GE Z， LIU S T， LI Z M， et al. OTA： Optimal transport assignment for object detection［C］∥2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2021： 303-312.
37	WANG C Y， MARK LIAO H Y， WU Y H， et al. CSPNet： A new backbone that can enhance learning capability of CNN［C］∥2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE Press， 2020： 1571-1580.
38	LIU S， QI L， QIN H F， et al. Path aggregation network for instance segmentation［C］∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2018： 8759-8768.
39	REZATOFIGHI H， TSOI N， GWAK J Y， et al. Generalized intersection over union： A metric and a loss for bounding box regression［C］∥2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE Press， 2019： 658-666.

特征提取组件	mAP/%
特征提取组件	MOT-FLY	DUT Anti-UAV Tracking
ResNet18	62.0	53.0
ResNet34	65.4	56.8
ResNet50	68.8	60.3
ResNet18-BE	62.7	58.9
ResNet34-BE	67.0	65.7
ResNet50-BE	69.6	66.8

关联算法	IDs	IDF1/%	MOTA/%
DeepSORT-ResNet18	534	43.0	42.0
DeepSORT-ResNet34	515	43.5	42.3
DeepSORT-ResNet50	499	44.1	42.5
DeepSORT-BEFE	329	48.3	44.0
Tracktor-ResNet18	114	41.1	32.9
Tracktor-ResNet34	111	41.3	33.4
Tracktor-ResNet50	106	41.0	33.9
Tracktor-BEFE	81	41.4	35.9

关联算法	局部区域半径/pixel	IDs	IDF1/%	MOTA/%
Byte-LGA	100	85	58.5	48.3
Byte-LGA	200	79	58.7	48.5
Byte-LGA	500	75	59.1	48.5
Byte-LGA	1 000	85	58.6	48.4
Byte-LGA	2 000	87	58.3	48.3

关联算法	检测组件	IDs	IDF1/%	MOTA/%
SORT	FRCNN	554	33.1	42.0
DeepSORT	FRCNN	499	44.1	42.5
Byte	FRCNN	115	58.3	48.0
SORT+LGA	FRCNN	263	37.7	44.6
DeepSORT+LGA	FRCNN	143	46.7	45.6
Byte+LGA	FRCNN	75	59.1	48.5

算法	检测组件	BEFE	LGA	IDs	IDF1/%	MOTA/%
Baseline	FRCNN			115	58.3	48.0
Baseline+BEFE	FRCNN	√		75	58.9	48.6
Baseline+BELGCA	FRCNN	√	√	57	59.0	48.9

Vision-based air-to-air multi-UAVs tracking

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 39

Related Articles 15

Recommended Articles

Metrics

Comments

算法	IDs	IDF1/%	MOTA/%	FPS
CenterTrack	48	38.2	36.6	39.2
CTracker	153	36.3	34.7	12.1
FairMOT	53	39.8	38.5	46.0
DeepSORT	708	41.0	43.0	44.4
ByteTrack	85	56.7	49.0	53.1
OC-SORT	58	53.0	50.2	53.5
BELGTracker	57	58.6	51.7	46.9

算法	FPS
BELGTracker-ResNet18-BE	28.0
BELGTracker-ResNet34-BE	27.2
BELGTracker-ResNet50-BE	24.5

[1]	Jianyu XU, Li ZHOU, Zhanxue WANG, Jie SHI, Hao SHI. Calculation method for hypersonic plume infrared radiation based on a fast line-by-line calculation model [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 630778-630778.
[2]	Lingjie MENG, Hongguang LI, Xinjun LI. SAR image simulation method guided by geomorphic category information [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(7): 331003-331003.
[3]	Zhihao ZHAO, Zhaohua YANG, Yun WU, Yuanjin YU. Single-photon counting imaging denoising method based on deep learning in low-light environment [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(3): 630531-630531.
[4]	Yiquan WU, Kang TONG. Research advances on deep learning-based small object detection in UAV aerial images [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(3): 30848-030848.
[5]	Chenhao ZHAO, Dewei WU, Jing HE, Qian WU. A semantic feature matching algorithm for UAV visual pose estimation [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(2): 330406-330406.
[6]	Yao LU, Zihao LI, Zhunga LIU, Yanbo YANG. Intelligent data association and tracking of different types of multiple targets based on Transformer [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(17): 331643-331643.
[7]	Xiaowei JIANG, Yiquan WU. Research progress of UAV aerial image mosaic methods [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(17): 331799-331799.
[8]	Jiang ZHAO, Minghao PI, Bailing TIAN, Pei CHI, Yingxun WANG. Self-organized consensus decision-making method for swarm UAV tracking multiple targets [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(16): 331635-331635.
[9]	Fang LIU, Chenyang LU, Yan LU, Xin WANG. Adaptive template update-based Transformer algorithm for UAV target tracking [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(16): 331687-331687.
[10]	Lin CHEN, Xiwen GU, Zhiying CHEN, Zhuo ZHANG, Xiaoliang SUN. High-precision monocular vision pose measurement for large distance span in carrier landing guidance [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(15): 331568-331568.
[11]	Haijun ZHANG, Qingyue XIA, Xu MA, Chao REN, Yang LU. A review of unmanned aerial vehicles deployment optimization in 6G low-altitude communication scenarios [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(11): 531296-531296.
[12]	Bin SUN, Hang YOU, Wenbo LI, Xiangrui LIU, Jiayi MA. Dual-band payload image fusion and its applications in low-altitude remote sensing [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(11): 531343-531343.
[13]	Wenxiao HU, Di MU, Zhi LI, Yingyi GUO, Xinmin CHEN. Key technical issues and innovation strategies for development of low-altitude economy [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(11): 531539-531539.
[14]	Fanteng MENG, Yong QIN, Jing CUI, Yunpeng WU, Zicheng ZHANG, Shaowei WEI. Unknown risk detection in external environment of railroad using UAV images [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(11): 531262-531262.
[15]	Weishi CHEN, Hongchuang NIU, Xin WANG, Jian WAN, Xianfeng LU, Jie ZHANG, Qingbin WANG. Review on multi-source detection technologies for birds and drones in airport clearance area [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(10): 31251-031251.