基于深度学习的无人机航拍视频多目标检测与跟踪研究进展

doi:10.7527/S1000-6893.2023.28334

Abstract

Abstract: With the increasing convenience of data acquisition for aerial photography of Unmanned Aerial Vehicle (UAV), the multi-target detection and tracking technology based on the UAV platform has developed rapidly and has broad application prospects in civil and military fields. In recent years, the rapid progress of in-depth learning has also provided a variety of more effective solutions. However, such challenging problems as sudden changes in the appearance of the target, serious occlusion of the target area, and disappearance and reappearance of the target from the perspective of UAV have not been completely solved. In this paper, we summarize the algorithms of multi-target detection and tracking in UAV aerial video based on depth learning, and summarize the latest progress in this field, including multi-target detection and multi-object tracking. The multi-object detection module is divided into two parts: Two-stage and One-stage. For the multi-object tracking module, according to the two classical frameworks of Tracking Based on Detection and Joint Detection Tracking, the principles of the two kinds of algorithms are described and their advantages and disadvantages are analyzed. Then, the existing public data sets are statistically analyzed, and the optimal schemes of the benchmark challenge VisDrone Challenge in the field of multi-target detection and tracking based on UAV aerial video in recent years are compared and analyzed. Finally, the paper summarizes the urgent problems of multi-object detection and tracking from the perspective of UAV and looks forward to the possible research directions in the future, providing a reference for the follow-up researchers.

Key words: UAV aerial video, Multi object detection, Multi target tracking, Deep learning, One stage detection, Two stage detection, Tracking Based on detection, Joint Detection Tracking

CLC Number:

V279
TP391

References

[1] 李玺,查宇飞,张天柱, 等. 深度学习的目标跟踪算法综述[J]. 中国图象图形学报, 2019, 24(12): 2057-2080. [2] ELHOSENY M. Multi-object detection and tracking (MODT) machine learning model for real-time video surveillance systems[J]. Circuits, Systems, and Signal Processing, 2020, 39(2): 611-630. [3] 徐芳,刘晶红,孙辉,等.光学遥感图像海面船舶目标检测技术进展[J]. 光学精密工程, 2021, 29(04): 916-931. [4] 江波,屈若锟,李彦冬,等.基于深度学习的无人机航拍目标检测研究综述[J].航空学报,2021,42(04):137-151. [5] AMMAR A, KOUBAA A, AHMED M, et al. Vehicle detection from aerial images using deep learning: A comparative study[J]. Electronics, 2021, 10(7): 820. [6] YAO H, QIN R, CHEN X. Unmanned aerial vehicle for remote sensing applications - a review[J]. Remote Sens-ing, 2019, 11(12): 1443. [7] RAVINDRAN R, SANTORA M J, JAMALI M M. Multi-object detection and tracking, based on DNN, for autonomous vehicles: A review[J]. IEEE Sensors Journal, 2020, 21(5): 5668-5677. [8] 葛宝义,左宪章,胡永江. 视觉目标跟踪方法研究综述[J]. 中国图象图形学报, 2018, 23(08): 1091-1107. [9] BOURAYA S, BELANGOUR A. Approaches to video real time multi-object tracking and object detection: A survey[C]//2021 12th International Symposium on Image and Signal Processing and Analysis. Piscataway: IEEE Press, 2021: 145-151. [10] PAL S K, PRAMANIK A, MAITI J, et al. Deep learning in multi-object detection and tracking: state of the art[J]. Applied Intelligence, 2021, 51(9): 6400-6429. [11] CIAPARRONE G, SáNCHEZ F L, TABIK S, et al. Deep learning in video multi-object tracking: A sur-vey[J]. Neurocomputing, 2020, 381: 61-88. [12] RANGESH A, TRIVEDI M M. No blind spots: full-surround multi-object tracking for autonomous vehicles using cameras and lidars[J]. IEEE Transactions on Intel-ligent Vehicles, 2019, 4(4): 588-599. [13] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and se-mantic segmentation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2014: 580-587. [14] GIRSHICK R. Fast R-CNN[C]//2015 IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2015: 1440-1448. [15] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. Advances in Neural Information Processing Systems, 2015(28): 1-9. [16] CAI Z, VASCONCELOS N. Cascade R-CNN: Delving into high quality object detection[C]//2018 IEEE Con-ference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 6154-6162. [17] LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single shot multibox detector[C]//2016 European Conference on Computer Vision. Springer, Cham, 2016: 21-37. [18] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detec-tion[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2016: 779-788. [19] REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 7263-7271. [20] REDMON J, FARHADI A. YOLO V3: An incremental improvement[DB/OL]. arXiv preprint:1804.02767, 2018. [21] BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLO V4: Optimal speed and accuracy of object detection [DB/OL]. arXiv preprint: 2004.10934, 2020. [22] WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLO V7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[DB/OL]. arXiv pre-print:2207.02696, 2022. [23] DUAN K, BAI S, XIE L, et al. Centernet: keypoint triplets for object detection[C]//2019 IEEE/CVF Inter-national Conference on Computer Vision. Piscataway: IEEE Press, 2019: 6569-6578. [24] CIAPARRONE G, SáNCHEZ F L, TABIK S, et al. Deep learning in video multi-object tracking: A sur-vey[J]. Neurocomputing, 2020(381): 61-88. [25] AYALEW A, POOJA D. A review on object detection from unmanned aerial vehicle using CNN[J]. Interna-tional Journal of Advance Research, Ideas and Innova-tions in Technology, 2019, 5: 241-243. [26] OSCO L P, JUNIOR J M, RAMOS A P M, et al. A review on deep learning in UAV remote sensing[J]. Inter-national Journal of Applied Earth Observation and Geoinformation, 2021, 102: 102456. [27] SRIVASTAVA S, NARAYAN S, MITTAL S. A survey of deep learning techniques for vehicle detection from UAV images[J]. Journal of Systems Architecture, 2021, 117: 102152. [28] CAZZATO D, CIMARELLI C, SANCHEZ-LOPEZ J L, et al. A survey of computer vision methods for 2d object detection from unmanned aerial vehicles[J]. Journal of Imaging, 2020, 6(8): 78. [29] 陈琳,刘允刚.面向无人机的视觉目标跟踪算法：综述与展望[J]. 信息与控制, 2022, 51(01): 23-40. [30] VIOLA P, JONES M. Rapid object detection using a boosted cascade of simple features[C]//2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2001, 1: 511-518. [31] DALAL N, TRIGGS B. Histograms of oriented gradi-ents for human detection[C]//2005 IEEE Computer So-ciety Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2005: 886-893. [32] FELZENSZWALB P, MCALLESTER D, RAMANAN D. A discriminatively trained, multiscale, deformable part model[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2008: 1-8. [33] ABUGHALIEH K M, SABABHA B H, RAWASHDEH N A. A video-based object detection and tracking system for weight sensitive UAVs[J]. Multimedia Tools and Applications, 2019, 78(7): 9149-9167. [34] BAYKARA H C, BIYIK E, GüL G, et al. Real-time detection, tracking and classification of multiple moving objects in UAV videos[C]//2017 IEEE 29th International Conference on Tools with Artificial Intelligence. Piscataway: IEEE Press, 2017: 945-950. [35] JIANG X, CAO X. Surveillance from above: A detec-tion-and-prediction based multiple target tracking method on aerial videos[C]//2016 Integrated Communications Navigation and Surveillance. Piscataway: IEEE Press, 2016:4D2-1-4D2-13. [36] AVOLA D, CINQUE L, DIKO A, et al. MS-Faster R-CNN: Multi-stream backbone for improved Faster R-CNN object detection and aerial tracking from UAV im-ages[J]. Remote Sensing, 2021, 13(9): 1670. [37] STADLER D. Multi-Object tracking in Drone Vide-os[C]//2020 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Labora-tory. Germay: IOSB, 2021:123-133. [38] HUANG W, ZHOU X, DONG M, et al. Multiple ob-jects tracking in the UAV system based on hierarchical deep high-resolution network[J]. Multimedia Tools and Applications, 2021, 80(9): 13911-13929. [39] ZHANG S, ZHUO L, ZHANG H, et al. Object tracking in unmanned aerial vehicle videos via multifeature dis-crimination and instance-aware attention network[J]. Remote Sensing, 2020, 12(16): 2646. [40] LIU Z, GAO G, SUN L, et al. HRDNet: high-resolution detection network for small objects[C]//2021 IEEE In-ternational Conference on Multimedia and Expo. Pisca-taway: IEEE Press, 2021: 1-6. [41] LIU Y, YANG F, HU P. Small-object detection in UAV-captured images via multi-branch parallel feature pyramid networks[J]. IEEE Access, 2020,8: 145740-145750. [42] AZIMI S M, KRAUS M, BAHMANYAR R, et al. Mul-tiple pedestrians and vehicles tracking in aerial imagery: a comprehensive study [DB/OL]. arXiv preprint: 2010.09689, 2020. [43] DU Y, WAN J, ZHAO Y, et al. GIAOTracker: A com-prehensive framework for MCMOT with global infor-mation and optimizing strategies in VisDrone 2021[C]//2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2021: 2809-2819. [44] T?TTRUP D, SKOVGAARD S L, SEJERSEN J F, et al. A fast and accurate approach to multiple-vehicle lo-calization and tracking from monocular aerial images[J]. Journal of Imaging, 2021, 7(12): 270. [45] ALBABA B M, OZER S. SyNet: An ensemble network for object detection in UAV images[C]//2020 25th In-ternational Conference on Pattern Recognition. Pisca-taway: IEEE Press, 2021: 10227-10234. [46] CAO J, CHOLAKKAL H, ANWER R M, et al. D2det: towards high quality object detection and instance seg-mentation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 11485-11494. [47] YANG F, FAN H, CHU P, et al. Clustered object detec-tion in aerial images[C]// 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 8311-8320. [48] ZHANG R, SHAO Z, HUANG X, et al. Object detec-tion in UAV images via global density fused convolu-tional network[J]. Remote Sensing, 2020, 12(19): 3140. [49] YU W, YANG T, CHEN C. Towards resolving the chal-lenge of long-tail distribution in UAV images for object detection[C]// 2021 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway: IEEE Press, 2021: 3258-3267. [50] ZHANG X, IZQUIERDO E, CHANDRAMOULI K. Dense and small object detection in UAV vision based on cascade network[C]//2019 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 118-126. [51] YANG J, XIE X, YANG W. Effective contexts for UAV vehicle detection[J]. IEEE Access, 2019, 7: 85042-85054. [52] WU Z, SURESH K, NARAYANAN P, et al. Delving into robust object detection from unmanned aerial vehi-cles: a deep nuisance disentanglement approach[C]//2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 1201-1210. [53] ZHANG W, LIU C, CHANG F, et al. Multi-scale and occlusion aware network for vehicle detection and seg-mentation on UAV aerial images[J]. Remote Sensing, 2020, 12(11): 1760. [54] CHEN C, ZHANG Y, LV Q, et al. Rrnet: a hybrid detec-tor for object detection in drone-captured imag-es[C]//2019 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 100-108. [55] WANG H, WANG Z, JIA M, et al. Spatial attention for multi-scale feature refinement for object detection[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 64-72. [56] TANG Z, LIU X, YANG B. PENet: Object detection using points estimation in high definition aerial images[C] //2020 19th IEEE International Conference on Machine Learning and Applications. Piscataway: IEEE Press, 2020: 392-398. [57] DIKE H U, ZHOU Y. A robust quadruplet and faster region-based CNN for UAV video-based multiple object tracking in crowded environment[J]. Electronics, 2021, 10(7): 795. [58] LIN Q, DING Y, XU H, et al. ECASCADE-RCNN: Enhanced cascade RCNN for multi-scale object detection in UAV images[C]//2021 7th International Conference on Automation, Robotics and Applications. Piscataway: IEEE Press, 2021: 268-272. [59] YOUSSEF Y, ELSHENAWY M. Automatic vehicle counting and tracking in aerial video feeds using cascade region-based convolutional neural networks and feature pyramid networks[J]. Transportation Research Record, 2021, 2675(8): 304-317. [60] LEE Y, LEE S, YOO J, et al. Efficient single-shot multi-object tracking for vehicles in traffic scenarios[J]. Sen-sors, 2021, 21(19): 6358. [61] MAKAROV S B, PAVLOV V A, BEZBORODOV A K, et al. Multiple object tracking using convolutional neural network on aerial imagery sequences[C]//2021 In-ternational Youth Conference on Electronics, Telecom-munications and Information Technologies. Springer, Cham, 2021: 413-420. [62] HOSSAIN S, LEE D. Deep learning-based real-time multiple-object detection and tracking from aerial imagery via a flying robot with GPU-based embedded devices[J]. Sensors, 2019, 19(15): 3371. [63] LI J, CHEN S, ZHANG F, et al. An adaptive framework for multi-vehicle ground speed estimation in airborne videos[J]. Remote Sensing, 2019, 11(10): 1241. [64] EMIYAH C, NYARKO K, CHAVIS C, et al. Extracting vehicle track information from unstabilized drone aerial videos using YOLO v4 common object detector and computer vision[C]//2021 Future Technologies Confer-ence. Springer, Cham, 2021: 232-239. [65] YANG T, LI D, BAI Y, et al. Multiple-object-tracking algorithm based on dense trajectory voting in aerial vid-eos[J]. Remote Sensing, 2019, 11(19): 2278. [66] ZHANG P, ZHONG Y, LI X. SlimYOLO V3: Narrower, faster and better for real-time UAV applications[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 37-45. [67] LIU M, WANG X, ZHOU A, et al. UAV-YOLO: small object detection on unmanned aerial vehicle perspec-tive[J]. Sensors, 2020, 20(8): 2238. [68] KYRKOU C, PLASTIRAS G, THEOCHARIDES T, et al. DroNet: efficient convolutional neural network de-tector for real-time UAV applications[C]//2018 Design, Automation & Test in Europe Conference & Exhibition. Piscataway: IEEE Press, 2018: 967-972. [69] LIU Y, YANG F, HU P. Small-object detection in UAV-captured images via multi-branch parallel feature pyramid networks[J]. IEEE Access, 2020, 8: 145740-145750. [70] BALAMURALIDHAR N, TILON S, NEX F. MultEYE: Monitoring system for real-time vehicle detection, tracking and speed estimation from UAV imagery on edge-computing platforms[J]. Remote sensing, 2021, 13(4): 573. [71] NING M, MA X, LU Y, et al. SeeFar: Vehicle Speed Estimation and Flow Analysis from a Moving UAV[C]// International Conference on Image Analysis and Pro-cessing. Springer, Cham, 2022: 278-289. [72] LI M, ZHAO X, LI J, et al. COMNet: Combinational neural network for object detection in UAV-borne thermal images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(8): 6662-6673. [73] ZHANG J, LIANG X, WANG M, et al. Coarse-to-fine object detection in unmanned aerial vehicle imagery using lightweight convolutional neural network and deep motion saliency[J]. Neurocomputing, 2020, 398: 555-565. [74] LIU Y, DING Z, CAO Y, et al. Multi-scale feature fusion UAV image object detection method based on dilated convolution and attention mechanism[C]//2020 8th International Conference on Information Technology: IoT and Smart City. New York: ACM, 2020: 125-132. [75] ZHANG Z, LIU Y, LIU T, et al. DAGN: A real-time UAV remote sensing image vehicle detection frame-work[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(11): 1884-1888. [76] LI Z, LIU X, ZHAO Y, et al. A lightweight multi-scale aggregated model for detecting aerial images captured by UAVs[J]. Journal of Visual Communication and Image Representation, 2021, 77: 103058. [77] JADHAV A, MUKHERJEE P, KAUSHIK V, et al. Aeri-al multi-object tracking by detection using deep associa-tion networks[C]//2020 National Conference on Com-munications. Piscataway: IEEE Press, 2020: 1-6. [78] 刘芳,浦昭辉,张帅超.基于注意力特征融合的无人机多目标跟踪算法[J/OL]. 控制与决策:1-9 [2022-09-27]. https://kns.cnki.net/kcms/detail/21.1124.tp.20211201.1147.016.html. [79] PI Z, LIAN Y, CHEN X, et al. A novel spatial and tem-poral context-aware approach for drone-based video ob-ject detection[C]//2019 IEEE/CVF International Con-ference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 179-188. [80] LIANG Z, WANG J, XIAO G, et al. FAANet: feature-aligned attention network for real-time multiple object tracking in UAV videos[J]. Chinese Optics Letters, 2022, 20(8): 081101. [81] LIANG X, ZHANG J, ZHUO L, et al. Small object detection in unmanned aerial vehicle images using feature fusion and scaling-based single shot detector with spatial context analysis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(6): 1758-1770. [82] ZHANG H, WANG G, LEI Z, et al. Eye in the sky: Drone-based object tracking and 3d localization[C]// 2019 27th ACM International Conference on Multimedia. New York: ACM, 2019: 899-907. [83] WANG X, LI W, GUO W, et al. SPB-YOLO: an effi-cient real-time detector for unmanned aerial vehicle im-ages[C]//2021 International Conference on Artificial In-telligence in Information and Communication. Piscataway: IEEE Press, 2021: 099-104. [84] WU Q, ZHOU Y. Real-time object detection based on unmanned aerial vehicle[C]//2019 IEEE 8th Data Driven Control and Learning Systems Conference. Piscataway: IEEE Press, 2019: 574-579. [85] WU P, XU H, DING Y, et al. An improved online mul-tiple object tracking algorithm based on KFHT motion compensation model in the aerial videos[C]// Seventh Symposium on Novel Photoelectronic Detection Tech-nology and Applications. Bellingham: SPIE, 2021, 11763: 2431-2436. [86] KAPANIA S, SAINI D, GOYAL S, et al. Multi object tracking with UAVs using deep SORT and YOLO V3 RetinaNet detection framework[C]//2020 1st ACM Workshop on Autonomous and Intelligent Mobile Sys-tems. New York: ACM, 2020(1): 1-6. [87] TIAN G, LIU J, YANG W. A dual neural network for object detection in UAV images[J]. Neurocomputing, 2021, 443: 292-301. [88] AL-SHAKARJI N M, BUNYAK F, SEETHARAMAN G, et al. Multi-object tracking cascade with multi-step data association and occlusion handling[C]//2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance. Piscataway: IEEE Press, 2018: 1-6. [89] WANG J, SIMEONOVA S, SHAHBAZI M. Orienta-tion-and scale-invariant multi-vehicle detection and tracking from unmanned aerial videos[J]. Remote Sens-ing, 2019, 11(18): 2155. [90] YU H, LI G, ZHANG W, et al. Self-balance motion and appearance model for multi-object tracking in UAV[C]//2019 ACM Multimedia Asia. New York: ACM, 2019(12):1-6 [91] AHN H, CHO H J. Research of multi-object detection and tracking using machine learning based on knowledge for video surveillance system[J]. Personal and Ubiquitous Computing, 2022(26): 385-394. [92] LUSK P C, BEARD R W. Visual multiple target tracking from a descending aerial platform[C]//2018 Annual American Control Conference. Piscataway: IEEE Press, 2018: 5088-5093. [93] LI W, MU J, LIU G. Multiple object tracking with mo-tion and appearance cues [C]//2019 IEEE/CVF Interna-tional Conference on Computer Vision Workshops. Pis-cataway: IEEE Press, 2019: 161-169. [94] ARDO H, NILSSON M. Multi target tracking from drones by learning from generalized graph differ-ences[C]//2019 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 46-54. [95] FU H, GUAN J, JING F, et al. A real-time multi-vehicle tracking framework in intelligent vehicular networks[J]. China Communications, 2021, 18(6): 89-99. [96] HE Y, FU C, LIN F, et al. Towards robust visual tracking for unmanned aerial vehicle with tri-attentional correlation filters[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2020: 1575-1582. [97] STADLER D, SOMMER L W, BEYERER J. Pas track-er: Position-, appearance-and size-aware multi-object tracking in drone videos[C]//2020 European Conference on Computer Vision. Springer, Cham, 2020: 604-620. [98] LI F, FU C, LIN F, et al. Training-set distillation for real-time UAV object tracking[C]//2020 IEEE International Conference on Robotics and Automation. Piscataway: IEEE Press, 2020: 9715-9721. [99] LI Y, FU C, HUANG Z, et al. Intermittent contextual learning for keyfilter-aware UAV object tracking using deep convolutional feature[J]. IEEE Transactions on Multimedia, 2020, 23: 810-822. [100] XU S, SAVVARIS A, HE S, et al. Real-time implemen-tation of YOLO+ JPDA for small scale UAV multiple object tracking[C]// 2018 International Conference on Unmanned Aircraft Systems. Piscataway: IEEE Press, 2018: 1336-1341. [101] LEE M H, YEOM S. Tracking of moving vehicles with a UAV[C]// 2018 International Symposium on Advanced Intelligent Systems; International Conference on Soft Computing and Intelligent Systems. Piscataway: IEEE Press,2018: 928-931. [102] 王旭辰,韩煜祺,唐林波,等.基于深度学习的无人机载平台多目标检测和跟踪算法研究[J]. 信号处理, 2022, 38(01): 157-163. [103] LUO X, ZHAO R, GAO X. Research on UAV multi-object tracking based on deep learning[C]//2021 IEEE International Conference on Networking, Sensing and Control. Piscataway: IEEE Press, 2021, 1: 1-6. [104] KHALKHALI M B, VAHEDIAN A, YAZDI H S. Situation assessment-augmented interactive kalman filter for multi-vehicle tracking[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(4): 3766-3776. [105] WOJKE N, BEWLEY A, PAULUS D. Simple online and realtime tracking with a deep association metric[C]// 2017 IEEE International Conference on Image Processing. Piscataway: IEEE Press, 2017: 3645-3649. [106] WU Y, WANG Y, ZHANG D, et al. Research on vehicle tracking method based on UAV video[C]//2022 Interna-tional Conference on Internet of Things and Smart City. Bellingham: SPIE, 2022, 12249: 801-806. [107] WU H, DU C, JI Z, et al. SORT-YM: An algorithm of multi-object tracking with YOLO V4-tiny and motion prediction[J]. Electronics, 2021, 10(18): 2319. [108] CHEN T, PENNISI A, LI Z, et al. A hierarchical associ-ation framework for multi-object tracking in airborne videos[J]. Remote Sensing, 2018, 10(9): 1347. [109] YU H, LI G, SU L, et al. Conditional GAN based indi-vidual and global motion fusion for multiple object tracking in UAV videos[J]. Pattern Recognition Letters, 2020, 131: 219-226. [110] PAN S, TONG Z, ZHAO Y, et al. Multi-object tracking hierarchically in visual data taken from drones[C]//2019 IEEE/CVF International Conference on Computer Vision Workshops. Piscataway: IEEE Press, 2019: 135-143. [111] BAHMANYAR R, AZIMI S, REINARTZ P. Multiple vehicle and people tracking in aerial imagery using stack of micro single-object-tracking CNNs[C]// 2019 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Tehran: ISPRS, 2019:163-170. [112] SAETCHNIKOV I, SKAKUN V, TCHERNIAVSKAIA E. Efficient objects tracking from an unmanned aerial vehicle[C]//2021 IEEE 8th International Workshop on Metrology for Aero Space. Piscataway: IEEE Press, 2021: 221-225. [113] MARVASTI-ZADEH S M, KHAGHANI J, GHANEI-YAKHDAN H, et al. COMET: context-aware IOU-guided network for small object tracking[DB/OL]. arXiv preprint:2006.02597, 2020 [114] LIN D, CHEN Q, ZHOU C, et al. TraSw: tracklet-switch adversarial attacks against multi-object tracking [DB/OL]. arXiv preprint: 2111.08954, 2021. [115] ROBICQUET A, SADEGHIAN A, ALAHI A, et al. Learning social etiquette: Human trajectory understanding in crowded scenes[C]// 2016 European Conference on Computer Vision. Springer, Cham, 2016: 549-565. [116] DU D, QI Y, YU H, et al. The unmanned aerial vehicle benchmark: Object detection and tracking[C]// 2018 Eu-ropean Conference on Computer Vision. Springer, Cham, 2018: 370-386. [117] ZHU P, WEN L, BIAN X, et al. Vision meets drones: a challenge [DB/OL]. arXiv preprint: 1804.07437, 2018. [118] ZHU P, WEN L, DU D, et al. Vision meets drones: past, present and future [DB/OL]. arXiv preprint: 2001.06303, 2020. [119] BONDI E, JAIN R, AGGRAWAL P, et al. BIRDSAI: a dataset for detection and tracking in aerial thermal infra-red videos[C]// 2020 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway: IEEE Press, 2020: 1747-1756. [120] HSIEH M R, LIN Y L, HSU W H. Drone-based object counting by spatially regularized regional proposal net-work[C]//2017 IEEE International Conference on Com-puter Vision. Piscataway: IEEE Press, 2017: 4145-4153. [121] XU X, ZHANG X, YU B, et al. Dac-sdc low power object detection challenge for UAV applications[J]. IEEE Transactions on Pattern Analysis and Machine In-telligence, 2019, 43(2): 392-403. [122] MANDAL M, KUMAR L K, VIPPARTHI S K. MOR-UAV: A benchmark dataset and baselines for moving object recognition in UAV videos[C]// 2020 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2626-2635. [123] ZHU P, SUN Y, WEN L, et al. Drone based rgbt vehicle detection and counting: a challenge[DB/OL]. arXiv preprint: 2003.02437. [124] BOZCAN I, KAYACAN E. Au-air: A multi-modal un-manned aerial vehicle dataset for low altitude traffic surveillance[C]//2020 IEEE International Conference on Robotics and Automation. Piscataway: IEEE Press, 2020: 8504-8510. [125] ZHANG H, SUN M, LI Q, et al. An empirical study of multi-scale object detection in high resolution UAV im-ages[J]. Neurocomputing, 2021, 421: 173-182. [126] ZHANG W, LIU C, CHANG F, et al. Multi-scale and occlusion aware network for vehicle detection and seg-mentation on UAV aerial images[J]. Remote Sensing, 2020, 12(11): 1760. [127] BERNARDIN K, STIEFELHAGEN R. Evaluating multiple object tracking performance: the clear mot metrics[J]. EURASIP Journal on Image and Video Pro-cessing, 2008, 2008: 1-10. [128] RISTANI E, SOLERA F, ZOU R, et al. Performance measures and a data set for multi-target, multi-camera tracking[C]// 2016 European conference on computer vision. Springer, Cham, 2016: 17-35. [129] LUITEN J, OSEP A, DENDORFER P, et al. HOTA: a higher order metric for evaluating multi-object tracking[J]. International Journal of Computer Vision, 2021, 129(2): 548-578.

Research progress of UAV aerial video multi-object detection and tracking based on deep learning

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	SU Lingfei, HUA Yongzhao, DONG Xiwang, REN Zhang. Human-UAV swarm multi-modal intelligent interaction methods [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 727001-727001.
[2]	ZHAO Liangyu, LI Dan, ZHAO Chenyue, JIANG Fei. Some achievements on detection methods of UAV autonomous landing markers [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 25882-025882.
[3]	CAO Ming, WANG Peng, ZUO Hongfu, ZENG Haijun, SUN Jianzhong, YANG Weidong, WEI Fang, CHEN Xuefeng. Current status, challenges and opportunities of civil aero-engine diagnostics & health management Ⅱ: Comprehensive off-board diagnosis, life management and intelligent condition based MRO [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 625574-625574.
[4]	LIU Jiaqi, FENG Yunwen, LU Cheng, XUE Xiaofeng, PAN Weihuang. Safety analysis of aero-engine operation based on intelligent neural network [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 625375-625375.
[5]	SHAO Yiwei, CHEN Jiayu, LIN Cuiying, WAN Cheng, GE Hongjuan, SHI Zhilong. Gearbox fault diagnosis with small training samples: An improved deep forest based method [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 625429-625429.
[6]	LIU Zichao, WANG Jiang, HE Shaoming, LI Yufei. A computational guidance algorithm for impact angle control based on predictor-corrector concept [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 325433-325433.
[7]	SUN Canfei, HUANG Linran, SHEN Yong. Fault diagnosis of helicopter planetary gear transmission system under complex operating conditions [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 625480-625480.
[8]	KANG Yuxiang, CHEN Guo, WEI Xunkai, ZHOU Lei. Deep residual hedging network and its application in fault diagnosis of rolling bearings [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 625201-625201.
[9]	LYU Shuaishuai, YANG Yu, WANG Binwen, YIN Chenfei. A crack identification method of aircraft structure based on improved FaceNet [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(6): 525463-525463.
[10]	LIU Tianyuan, ZHENG Hangbin, YANG Changqi, BAO Jinsong, WANG Junliang, GU Jun. Explainable deep learning method for laser welding defect detection [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(4): 524961-524961.
[11]	KANG Shuo, KE Zhenzheng, WANG Xuan, ZHU Weidong. Detection method of defects in automatic fiber placement based on fusion of infrared and visible images [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(3): 425187-425187.
[12]	ZHAN Qingliang, BAI Chunjin, ZHANG Ning, GE Yaojun. Feature extraction method of flow around airfoil based on time-history convolutional autoencoder [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(11): 526531-526531.
[13]	YI Jianmiao, DENG Feng, QIN Ning, LIU Xueqiang. Fast prediction of transonic flow field using deep learning method [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(11): 526747-526747.
[14]	CAI Guobiao, ZHANG Baiyi, HE Bijiao, WENG Huiyan, LIU Lihui. Intelligent computation of vacuum plume [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(10): 527352-527352.
[15]	LI Hongguang, WANG Fei, DING Wenrui. Survey on multi-task learning for object classification and recognition [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(1): 24889-024889.