基于导向强化Q学习的无人机路径规划

doi:10.7527/S1000-6893.2021.25109

Abstract

Abstract: With the increasing application of the Unmanned Aerial Vehicle (UAV) technology, the energy consumption and computing capacity of UAV are faced with bottleneck problems, so path planning of UAV is becoming increasingly important. In many cases, the UAV cannot obtain the exact location of the target point and environmental information in advance, and thus is difficult to plan an effective flight path. To solve this problem, this paper proposes a path planning method for UAV using the guided enhancement Q-learning algorithm. This method uses Receiving Signal Strength (RSS) to define the reward value, and continuously optimizes the path by using the Q-learning algorithm. The principle of "guided reinforcement" is proposed to accelerate the convergence speed of the Q learning algorithm. The simulation results show that the method proposed can realize autonomous navigation and fast path planning for UAV. Compared with the traditional algorithm, it can greatly reduce the number of iterations and obtain a shorter planned path.

Key words: UAV, path planning, receiving signal strength, Q-learning algorithm, guided enhancement

CLC Number:

V279

ZHOU Bin, GUO Yan, LI Ning, ZHONG Xijian. Path planning of UAV using guided enhancement Q-learning algorithm[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(9): 325109-325109.

References

[1] BECERRA V M. Autonomous control of unmanned aerial vehicles[J]. Electronics, 2019, 8(4):452.
[2] COUTINHO W P, BATTARRA M, FLIEGE J. The unmanned aerial vehicle routing and trajectory optimisation problem, a taxonomic review[J]. Computers & Industrial Engineering, 2018, 120:116-128.
[3] XU Y P, CHE C. A brief review of the intelligent algorithm for traveling salesman problem in UAV route planning[C]//2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC). Piscataway:IEEE Press, 2019:1-7.
[4] ZHAO Y J, ZHENG Z, ZHANG X Y, et al. Q learning algorithm based UAV path learning and obstacle avoidance approach[C]//201736th Chinese Control Conference (CCC). Piscataway:IEEE Press, 2017:3397-3402.
[5] CAI Y Z, XI Q B, XING X J, et al. Path planning for UAV tracking target based on improved A-star algorithm[C]//20191st International Conference on Industrial Artificial Intelligence (IAI). Piscataway:IEEE Press, 2019:1-6.
[6] CHEN X, CHEN X M. The UAV dynamic path planning algorithm research based on Voronoi diagram[C]//The 26th Chinese Control and Decision Conference (2014 CCDC). Piscataway:IEEE Press, 2014:1069-1071.
[7] LI W H. An improved artificial potential field method based on chaos theory for UAV route planning[C]//201934rd Youth Academic Annual Conference of Chinese Association of Automation (YAC). Piscataway:IEEE Press, 2019:47-51.
[8] AGGARWAL S, KUMAR N. Path planning techniques for unmanned aerial vehicles:A review, solutions, and challenges[J]. Computer Communications, 2020, 149:270-299.
[9] WAI R J, PRASETIA A S. Adaptive neural network control and optimal path planning of UAV surveillance system with energy consumption prediction[J]. IEEE Access, 2019, 7:126137-126153.
[10] SALAMAT B, TONELLO A M. A modelling approach to generate representative UAV trajectories using PSO[C]//201927th European Signal Processing Conference (EUSIPCO). Piscataway:IEEE Press, 2019:1-5.
[11] VILLANUEVA A, FAJARDO A. Deep reinforcement learning with noise injection for UAV path planning[C]//2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS). Piscataway:IEEE Press, 2019:1-6.
[12] PENG Z H, LI B, CHEN X T, et al. Online route planning for UAV based on model predictive control and particle swarm optimization algorithm[C]//Proceedings of the 10th World Congress on Intelligent Control and Automation. Piscataway:IEEE Press, 2012:397-401.
[13] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540):529-533.
[14] LI R X, FU L, WANG L L, et al. Improved Q-learning based route planning method for UAVs in unknown environment[C]//2019 IEEE 15th International Conference on Control and Automation (ICCA). Piscataway:IEEE Press, 2019:118-123.
[15] LI S D, XU X, ZUO L. Dynamic path planning of a mobile robot with improved Q-learning algorithm[C]//2015 IEEE International Conference on Information and Automation. Piscataway:IEEE Press, 2015:409-414.
[16] PHAM H X, LA H M, FEIL-SEIFER D, et al. Reinforcement learning for autonomous UAV navigation using function approximation[C]//2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Piscataway:IEEE Press, 2018:1-6.
[17] PENG J S. Mobile robot path planning based on improved Q-learning algorithm[J]. International Journal of Multimedia and Ubiquitous Engineering, 2015, 10(7):285-294.
[18] AL-HOURANI A, KANDEEPAN S, JAMALIPOUR A. Modeling air-to-ground path loss for low altitude platforms in urban environments[C]//2014 IEEE Global Communications Conference. Piscataway:IEEE Press, 2014:2898-2904.
[19] WATKINS C J C H, DAYAN P. Q-learning[J]. Machine Learning, 1992, 8(3-4):279-292.
[20] WANG Z Y, SHI Z G, LI Y K, et al. The optimization of path planning for multi-robot system using Boltzmann Policy based Q-learning algorithm[C]//2013 IEEE International Conference on Robotics and Biomimetics (ROBIO). Piscataway:IEEE Press, 2013:1199-1204.
[21] RASTOGI K, LEE J, HAREL-CANADA F, et al. Is Q-learning provably efficient? An extended analysis[DB/OL]. arXiv:2009.10396,2020.
[22] 陈崚, 孙海鹰. 蚁群算法一阶欺骗性问题的时间复杂度分析[J]. 模式识别与人工智能, 2010, 23(1):1-6. CHEN L, SUN H Y. Time complexity analysis of ant colony algorithm on first order deceptive problem[J]. Pattern Recognition and Artificial Intelligence, 2010, 23(1):1-6(in Chinese).
[23] HE X F, YU W, XU H S, et al. Towards 3D deployment of UAV base stations in uneven terrain[C]//201827th International Conference on Computer Communication and Networks (ICCCN).Piscataway:IEEE Press, 2018:1-9.

Path planning of UAV using guided enhancement Q-learning algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Haifeng WANG, Kunpeng LIU, Hongxin JIANG, Chenxi DU. Aerodynamic optimization method of propeller multi⁃design points and variable pitch angle strategy [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(9): 528831-528831.
[2]	Jing ZHAO, Dan SONG. Integrity monitoring method for GNSS/IMU integrated navigation system of UAV [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(7): 328943-328943.
[3]	Chuanyun WANG, Yang SU, Linlin WANG, Tian WANG, Jingjing WANG, Qian GAO. Multi-object continuous robust tracking algorithm for anti-UAV swarm [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(7): 329017-329017.
[4]	Hongyu YIN, Yu WU, Tianjiao LIANG. Cooperative path planning for patrol coverage of fixed wing UAV [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 328944-328944.
[5]	Xudong LUO, Yiquan WU, Jinlin CHEN. Research progress on deep learning methods for object detection and semantic segmentation in UAV aerial images [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 28822-028822.
[6]	Chuihuan KONG, Dawei WU, Zhaoguang TAN, Lijun PAN, Rubing MA, Jiangtao SI. Design of fully electric scheme for three⁃surface verification aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(6): 629618-629618.
[7]	Guangjia LI, Hongbo WANG, Kai ZHANG, Zhisheng YI. Lift enhancement and drag reduction technologies of solar powered unmanned aerial vehicles in near space: Review [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529644-529644.
[8]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.
[9]	Zhu WANG, Mengtong ZHANG, Zhenpeng ZHANG, Guangtong XU. Multi-UAV cooperative path planning based on multi-index dynamic priority [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328816-328816.
[10]	Gaojie ZHENG, Xiaoming HE, Dongpo LI, Huijun TAN, Kun WANG, Zhenlong WU, Depeng WANG. Double 90° deflection inlet/volute coupling flow characteristics of tail-powered unmanned aerial vehicle [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 128782-128782.
[11]	Hongzhen GUO, Mou CHEN, Yongdong DAI, Maofei WANG. Distributed adaptive event⁃triggered formation control for QUAVs [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729917-729917.
[12]	Kunda LIU, Xueming LIU, Bo ZHU, Qingrui ZHANG. Robust safe control for multi⁃UAV formation flight through narrow corridors [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729768-729768.
[13]	Wenkang HAO, Suyan BAO, Qifeng CHEN. Distributed control of UAVs formation based on port⁃Hamiltonian system [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729868-729868.
[14]	Xuejian WANG, Yongming WEN, Xiaorong SHI, Ningning ZHANG, Jiexi LIU. Design of hybrid intelligent decision framework for multi⁃agent and multi⁃coupling tasks [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729770-729770.
[15]	Yuqi CAO, Haoran FU, Fei GAO, Ximin LYU. Trajectory tracking control algorithm for canard⁃equipped tail⁃sitting vertical takeoff and landing UAV based on MPCC [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729950-729950.