基于导向强化Q学习的无人机路径规划

doi:10.7527/S1000-6893.2021.25109

Abstract

Abstract: With the increasing application of the Unmanned Aerial Vehicle (UAV) technology, the energy consumption and computing capacity of UAV are faced with bottleneck problems, so path planning of UAV is becoming increasingly important. In many cases, the UAV cannot obtain the exact location of the target point and environmental information in advance, and thus is difficult to plan an effective flight path. To solve this problem, this paper proposes a path planning method for UAV using the guided enhancement Q-learning algorithm. This method uses Receiving Signal Strength (RSS) to define the reward value, and continuously optimizes the path by using the Q-learning algorithm. The principle of "guided reinforcement" is proposed to accelerate the convergence speed of the Q learning algorithm. The simulation results show that the method proposed can realize autonomous navigation and fast path planning for UAV. Compared with the traditional algorithm, it can greatly reduce the number of iterations and obtain a shorter planned path.

Key words: UAV, path planning, receiving signal strength, Q-learning algorithm, guided enhancement

CLC Number:

V279

ZHOU Bin, GUO Yan, LI Ning, ZHONG Xijian. Path planning of UAV using guided enhancement Q-learning algorithm[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(9): 325109-325109.

References

[1] BECERRA V M. Autonomous control of unmanned aerial vehicles[J]. Electronics, 2019, 8(4):452.
[2] COUTINHO W P, BATTARRA M, FLIEGE J. The unmanned aerial vehicle routing and trajectory optimisation problem, a taxonomic review[J]. Computers & Industrial Engineering, 2018, 120:116-128.
[3] XU Y P, CHE C. A brief review of the intelligent algorithm for traveling salesman problem in UAV route planning[C]//2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC). Piscataway:IEEE Press, 2019:1-7.
[4] ZHAO Y J, ZHENG Z, ZHANG X Y, et al. Q learning algorithm based UAV path learning and obstacle avoidance approach[C]//201736th Chinese Control Conference (CCC). Piscataway:IEEE Press, 2017:3397-3402.
[5] CAI Y Z, XI Q B, XING X J, et al. Path planning for UAV tracking target based on improved A-star algorithm[C]//20191st International Conference on Industrial Artificial Intelligence (IAI). Piscataway:IEEE Press, 2019:1-6.
[6] CHEN X, CHEN X M. The UAV dynamic path planning algorithm research based on Voronoi diagram[C]//The 26th Chinese Control and Decision Conference (2014 CCDC). Piscataway:IEEE Press, 2014:1069-1071.
[7] LI W H. An improved artificial potential field method based on chaos theory for UAV route planning[C]//201934rd Youth Academic Annual Conference of Chinese Association of Automation (YAC). Piscataway:IEEE Press, 2019:47-51.
[8] AGGARWAL S, KUMAR N. Path planning techniques for unmanned aerial vehicles:A review, solutions, and challenges[J]. Computer Communications, 2020, 149:270-299.
[9] WAI R J, PRASETIA A S. Adaptive neural network control and optimal path planning of UAV surveillance system with energy consumption prediction[J]. IEEE Access, 2019, 7:126137-126153.
[10] SALAMAT B, TONELLO A M. A modelling approach to generate representative UAV trajectories using PSO[C]//201927th European Signal Processing Conference (EUSIPCO). Piscataway:IEEE Press, 2019:1-5.
[11] VILLANUEVA A, FAJARDO A. Deep reinforcement learning with noise injection for UAV path planning[C]//2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS). Piscataway:IEEE Press, 2019:1-6.
[12] PENG Z H, LI B, CHEN X T, et al. Online route planning for UAV based on model predictive control and particle swarm optimization algorithm[C]//Proceedings of the 10th World Congress on Intelligent Control and Automation. Piscataway:IEEE Press, 2012:397-401.
[13] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540):529-533.
[14] LI R X, FU L, WANG L L, et al. Improved Q-learning based route planning method for UAVs in unknown environment[C]//2019 IEEE 15th International Conference on Control and Automation (ICCA). Piscataway:IEEE Press, 2019:118-123.
[15] LI S D, XU X, ZUO L. Dynamic path planning of a mobile robot with improved Q-learning algorithm[C]//2015 IEEE International Conference on Information and Automation. Piscataway:IEEE Press, 2015:409-414.
[16] PHAM H X, LA H M, FEIL-SEIFER D, et al. Reinforcement learning for autonomous UAV navigation using function approximation[C]//2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Piscataway:IEEE Press, 2018:1-6.
[17] PENG J S. Mobile robot path planning based on improved Q-learning algorithm[J]. International Journal of Multimedia and Ubiquitous Engineering, 2015, 10(7):285-294.
[18] AL-HOURANI A, KANDEEPAN S, JAMALIPOUR A. Modeling air-to-ground path loss for low altitude platforms in urban environments[C]//2014 IEEE Global Communications Conference. Piscataway:IEEE Press, 2014:2898-2904.
[19] WATKINS C J C H, DAYAN P. Q-learning[J]. Machine Learning, 1992, 8(3-4):279-292.
[20] WANG Z Y, SHI Z G, LI Y K, et al. The optimization of path planning for multi-robot system using Boltzmann Policy based Q-learning algorithm[C]//2013 IEEE International Conference on Robotics and Biomimetics (ROBIO). Piscataway:IEEE Press, 2013:1199-1204.
[21] RASTOGI K, LEE J, HAREL-CANADA F, et al. Is Q-learning provably efficient? An extended analysis[DB/OL]. arXiv:2009.10396,2020.
[22] 陈崚, 孙海鹰. 蚁群算法一阶欺骗性问题的时间复杂度分析[J]. 模式识别与人工智能, 2010, 23(1):1-6. CHEN L, SUN H Y. Time complexity analysis of ant colony algorithm on first order deceptive problem[J]. Pattern Recognition and Artificial Intelligence, 2010, 23(1):1-6(in Chinese).
[23] HE X F, YU W, XU H S, et al. Towards 3D deployment of UAV base stations in uneven terrain[C]//201827th International Conference on Computer Communication and Networks (ICCCN).Piscataway:IEEE Press, 2018:1-9.

Path planning of UAV using guided enhancement Q-learning algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Qilei GUO, Weimin SANG, Junjie NIU, Ye YUAN. UAV flight strategy considering icing risk under complex meteorological conditions [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(1): 627518-627518.
[2]	HU Yangxiu, ZHAO Changchun, JIA Chenglong, QIAN Zhouyuan, HU Tao. Synchronous path formation control of UAV swarm based on robot operating system (ROS) [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726914-726914.
[3]	SU Lingfei, HUA Yongzhao, DONG Xiwang, REN Zhang. Human-UAV swarm multi-modal intelligent interaction methods [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 727001-727001.
[4]	WANG Xiaoyue, WANG Xun, WANG Yongzhen, FEI Teng, LIU Dawei. Evaluation method for combat effectiveness of task-based simulated UAV swarm system [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726937-726937.
[5]	LIU Lei, LIU Dawei, WANG Xiaoguang, CHEN Junnan, LIU Dongxing. Development status and outlook of UAV clusters and anti-UAV clusters [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726908-726908.
[6]	YANG Mingyue, SHOU Yingxin, TANG Yong, LIU Chang, XU Bin. Multi-quadrotor UAVs formation maintenance and collision avoidance control [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726913-726913.
[7]	ZHOU Wei, MA Peiyang, GUO Zheng, WANG Daoping, ZHOU Ruisun. Research of combined fixed-wing UAV based on wingtip chained [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 325946-325946.
[8]	LI Hui, LONG Teng, SUN Jingliang, XU Guangtong. Adaptive line-of-sight method for 3D path following of UAVs [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 326105-326105.
[9]	LU Fangxu, MI Zhichao, MA Jun, LI Aijing, WANG Hai. Optimal 3D placement of multi-QoS UAV base station based on potential game for communications [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 326137-326137.
[10]	ZHAO Liangyu, LI Dan, ZHAO Chenyue, JIANG Fei. Some achievements on detection methods of UAV autonomous landing markers [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 25882-025882.
[11]	ZHANG Zhouyu, CAO Yunfeng, FAN Yanming. Research progress of vision based aerospace conflict sensing technologies for small unmanned aerial vehicle in low altitude [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 25645-025645.
[12]	WANG Rui, ZHOU Zhou, GUO Ronghua, HUANG Yuechen. Multi-body dynamics simulation and experiment of solar-powered UAV parachute landing [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 225721-225721.
[13]	TANG Li, HAO Peng, REN Peige, ZHANG Zuyao, HE Xiang, ZHANG Xuejun. UAV abnormal behavior detection based on improved iForest algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 326789-326789.
[14]	ZHANG Honghong, GAN Xusheng, SUN Jingjuan, ZHAO Guhao, HAN Baohua. Analysis of low altitude UAV conflict resolution safety based on STPA-TOPAZ [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(7): 325354-325354.
[15]	LIU Fang, SUN Yanan. UAV target tracking algorithm based on adaptive fusion network [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(7): 325522-325522.