基于深度强化学习的固定翼无人机编队协调控制方法

doi:10.7527/S1000-6893.2020.24009

Abstract

Abstract: Due to the complexity of kinematics and environmental dynamics, controlling a squad of fixed-wing Unmanned Aerial Vehicles (UAVs) remains a challenging problem. Considering the uncertainty of complex and dynamic environments, this paper solves the coordination control problem of UAV formation based on the model-free deep reinforcement learning algorithm. A new action selection strategy, ε-imitation strategy, is proposed by combining the ε-greedy strategy and the imitation strategy to balance the exploration and the exploitation. Based on this strategy, the double Q-learning technique, and the dueling architecture, the ID3QN (Imitative Dueling Double Deep Q-Network) algorithm is developed to boost learning efficiency. The results of the Hardware-In-Loop experiments conducted in a high-fidelity semi-physical simulation system demonstrate the adaptability and practicality of the proposed ID3QN coordinated control algorithm.

Key words: fixed-wing UAVs, UAV formation, coordination control, deep reinforcement learning, neural networks

CLC Number:

V249.1
V279

XIANG Xiaojia, YAN Chao, WANG Chang, YIN Dong. Coordination control method for fixed-wing UAV formation through deep reinforcement learning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524009-524009.

References

[1] 宗群, 王丹丹, 邵士凯, 等. 多无人机协同编队飞行控制研究现状及发展[J]. 哈尔滨工业大学学报, 2017, 49(3):1-14. ZONG Q, WANG D D, SHAO S K, et al. Research status and development of multi UAV coordinated formation flight control[J]. Journal of Harbin Institute of Technology, 2017, 49(3):1-14(in Chinese).
[2] 樊琼剑, 杨忠, 方挺, 等. 多无人机协同编队飞行控制的研究现状[J]. 航空学报, 2009, 30(4):683-691. FAN Q J, YANG Z, FANG T, et al. Research status of coordinated formation flight control for multi-UAVs[J]. Acta Aeronautica et Astronautica Sinica, 2009, 30(4):683-691(in Chinese).
[3] 王祥科, 刘志宏, 丛一睿, 等. 小型固定翼无人机集群综述和未来发展[J]. 航空学报, 2020, 41(4):323732. WANG X K, LIU Z H, CONG Y R, et al. Miniature fixed-wing UAV swarms:Survey and directions[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(4):323732(in Chinese).
[4] 贾永楠, 田似营, 李擎. 无人机集群研究进展综述[J]. 航空学报, 2020, 41(S1):723738. JIA Y N, TIAN S Y, LI Q. The development of unmanned aerial vehicle swarms[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(S1):723738(in Chinese).
[5] KURIKI Y, NAMERIKAWA T. Formation control with collision avoidance for a multi-UAV system using decentralized MPC and consensus-based control[J]. Journal of Control, Measurement, and System Integration, 2015, 8(4):285-294.
[6] SAIF O, FANTONI I, ZAVALA-RIO A. Distributed integral control of multiple UAVs:Precise flocking and navigation[J]. IET Control Theory & Applications, 2019, 13(13):2008-2017.
[7] PHAM H X, LA H M, FEIL-SEIFER D, et al. Cooperative and distributed reinforcement learning of drones for field coverage[JDB/OL]. arXiv preprint:1803.07250,2018.
[8] HUNG S M, GIVIGI S N. A Q-learning approach to flocking with UAVs in a stochastic environment[J]. IEEE Transactions on Cybernetics, 2017, 47(1):186-197.
[9] SUTTON R S, BARTO A G. Reinforcement learning:An introduction[M]. Cambridge:MIT Press, 1998.
[10] 高阳, 陈世福, 陆鑫. 强化学习研究综述[J]. 自动化学报, 2004, 30(1):86-100. GAO Y, CHEN S F, LU X. Research on reinforcement learning:A review[J]. Acta Automatic Sinica, 2004, 30(1):86-100(in Chinese).
[11] YAN C, XIANG X. A path planning algorithm for UAV based on improved Q-learning[C]//International Conference on Robotics and Automation Sciences (ICRAS). Piscataway:IEEE Press, 2018:1-5.
[12] EVERETT M, CHEN Y F, HOW J P. Motion planning among dynamic, decision-making agents with deep reinforcement learning[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway:IEEE Press, 2018:3052-3059.
[13] TAI L, PAOLO G, LIU M. Virtual-to-real deep reinforcement learning:Continuous control of mobile robots for mapless navigation[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway:IEEE Press, 2017:31-36.
[14] LONG P, LIU W, PAN J. Deep-learned collision avoidance policy for distributed multiagent navigation[J]. IEEE Robotics and Automation Letters, 2017, 2(2):656-663.
[15] TOMIMASU M, MORIHIRO K, NISHIMURA H, et al. A reinforcement learning scheme of adaptive flocking behavior[C]//International Symposium on Artificial Life and Robotics (AROB). Oita:ISAROB, 2005:GS1-4.
[16] MORIHIRO K, ISOKAWA T, NISHIMURA H, et al. Characteristics of flocking behavior model by reinforcement learning scheme[C]//SICE-ICASE International Joint Conference. Piscataway:IEEE Press, 2006:4551-4556.
[17] LA H M, SHENG W. Distributed sensor fusion for scalar field mapping using mobile sensor networks[J]. IEEE Transactions on Cybernetics, 2013, 43(2):766-778.
[18] LA H M, LIM R, SHENG W. Multirobot cooperative learning for predator avoidance[J]. IEEE Transactions on Control Systems Technology, 2015, 23(1):52-63.
[19] WANG C, WANG J, ZHANG X, et al. A deep reinforcement learning approach to flocking and navigation of UAVs in large-scale complex environments[C]//IEEE Global Conference on Signal and Information Processing (GlobalSIP). Piscataway:IEEE Press, 2018:1228-1232.
[20] HUNG S M, GIVIGI S N, NOURELDIN A. A dyna-Q(λ) approach to flocking with fixed-wing UAVs in a stochastic environment[C]//IEEE International Conference on Systems, Man, and Cybernetics. Piscataway:IEEE Press, 2015:1918-1923.
[21] 左家亮, 杨任农, 张滢, 等. 基于启发式强化学习的空战机动智能决策[J]. 航空学报, 2017, 38(10):321168. ZUO J L, YANG R N, ZHANG Y, et al. Intelligent decision-making in air combat maneuvering based on heuristic reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2017, 38(10):321168(in Chinese).
[22] WATKINS C, DAYAN P. Q-learning[J]. Machine Learning, 1992, 8(3):279-292.
[23] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540):529-533.
[24] VAN HASSELT H. Double Q-learning[C]//Advances in Neural Information Processing Systems. Vancouver:MIT Press, 2010:2613-2621.
[25] VAN HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[C]//AAAI Conference on Artificial Intelligence. Menlo Park:AAAI, 2016:2094-2100.
[26] WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//International Conference on Machine Learning (ICML). Brookline:JMLR, 2016:1995-2003.
[27] WANG C, YAN C, XIANG X, et al. A continuous actor-critic reinforcement learning approach to flocking with fixed-wing UAVs[C]//Asian Conference on Machine Learning (ACML). Brookline:JMLR, 2019:64-79.
[28] QUINTERO S A P, COLLINS G E, HESPANHA J P. Flocking with fixed-wing UAVs for distributed sensing:A stochastic optimal control approach[C]//American Control Conference. Piscataway:IEEE Press, 2013:2025-2031.
[29] YAN C, XIANG X, WANG C. Towards real-time path planning through deep reinforcement learning for a UAV in dynamic environments[J]. Journal of Intelligent & Robotic Systems, 2020, 98:297-309.
[30] LV L, ZHANG S, DING D, et al. Path planning via an improved DQN-based learning policy[J]. IEEE Access, 2019, 7:67319-67330.
[31] NAIR V, HINTON G E. Rectified linear units improve restricted boltzmann machines[C]//International Conference on Machine Learning (ICML). Brookline:JMLR, 2010:807-814.

Coordination control method for fixed-wing UAV formation through deep reinforcement learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	LIU Jiaqi, FENG Yunwen, LU Cheng, XUE Xiaofeng, PAN Weihuang. Safety analysis of aero-engine operation based on intelligent neural network [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 625375-625375.
[2]	FU Xiaowei, WANG Hui, XU Zhe. Cooperative pursuit strategy for multi-UAVs based on DE-MADDPG algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(5): 325311-325311.
[3]	CAI Guobiao, ZHANG Baiyi, HE Bijiao, WENG Huiyan, LIU Lihui. Intelligent computation of vacuum plume [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(10): 527352-527352.
[4]	WANG Hui, JIA Zikai, JIN Ren, LIN Defu, FAN Junfang, XU Chao. Cooperative object detection in UAV-based vision-guided docking [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(1): 324854-324854.
[5]	XIA Peng, YANG Te, XU Jiang, WANG Le, YANG Zhichun. Reversed time sequence dynamic load identification method using time delay neural network [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(7): 224452-224452.
[6]	LI Hongguang, YU Ruonan, DING Wenrui. Research development of small object traching based on deep learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(7): 24691-024691.
[7]	DONG Wenqi, HE Feng. Hierarchical and distributed generation of information interaction topology for large scale UAV formation [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(6): 324380-324380.
[8]	DING Wenrui, LIU Chunlei, LI Yue, ZHANG Baochang. Binary convolutional neural network: Review [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(6): 24504-024504.
[9]	WANG Zelin, JI Ritian, HUI Xinyu, DING Chen, WANG Hui, BAI Junqiang. L-shaped directional heat transfer based on deep learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(6): 124242-124242.
[10]	LI Zuobiao, WEN Fengbo, TANG Xiaolei, SU Liangjun, WANG Songtao. Prediction of single-row hole film cooling performance based on deep learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524331-524331.
[11]	JIANG Bo, QU Ruokun, LI Yandong, LI Chenglong. Object detection in UAV imagery based on deep learning: Review [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524519-524519.
[12]	ZHANG Zhichao, GAO Taiyuan, ZHANG Lei, TUO Shuangfen. Aeroheating agent model based on radial basis function neural network [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524167-524167.
[13]	HUANG Shouqing, LIU Shouwen, ZHAI Baichen, ZHOU Yuan, HUANG Xiaokai, QIN Taichun. Prediction of CMG failure boundary domain based on combined stress test and neural network [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524208-524208.
[14]	FENG Yunwen, PAN Weihuang, LIU Jiaqi, LU Cheng, XUE Xiaofeng, LENG Jiaxing. Operational reliability of aircraft power plant based on machine learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524732-524732.
[15]	REN Feng, GAO Chuanqiang, TANG Hui. Machine learning for flow control: Applications and development trends [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524686-524686.