非对称机动能力多无人机智能协同攻防对抗

doi:10.7527/S1000-6893.2020.24152

Abstract

Abstract: The attack-defense game is an important combat scenario of future military Unmanned Aerial Vehicles (UAVs). This paper studies an attack-defense game between groups of UAVs with different maneuverability, establishing a multi-UAV cooperative attack and defense evolution model. Based on the multi-agent reinforcement learning theory, the autonomous decision-making method of multi-UAV cooperative attack-defense game is studied, and a centralized critic and distributed actor algorithm structure is proposed based on the actor-critic algorithm, guaranteeing the convergence of the algorithm and improving the efficiency of decision-making. The critic module of UAVs uses the global information to evaluate the decision-making quality during training, while the actor module only needs to rely on the local perception information to make autonomous decisions during execution, hence improving the effectiveness of the multi-UAV attack-defense game. The simulation results show that the proposed multi-UAV reinforcement learning method has a strong self-evolution property, endowing the UAV certain intelligence, that is, the stable autonomous learning ability. Through continuous training, the UAVs can autonomously learn cooperative attack or defense policies to improve the effectiveness of decision-making.

Key words: multi-UAV coordination, attack-defense games, reinforcement learning, centralized critic, distributed actors

CLC Number:

V279

CHEN Can, MO Li, ZHENG Duo, CHENG Ziheng, LIN Defu. Cooperative attack-defense game of multiple UAVs with asymmetric maneuverability[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(12): 324152-324152.

References

[1] ISAACS R. Differential games a mathematical theory with applications to warfare and pursuit, control and optimization[J]. Physics Bulletin, 1966, 17(2):1-2.
[2] BERKOVITZ L D. A variational approach to differential games[J]. Annals of Math Study, 1964, 127(52):127-174.
[3] HO Y, BRYSON A, BARON S. Differential games and optimal pursuit-evasion strategies[J]. IEEE Transactions on Automatic Control, 1965, 10(4):37-40.
[4] KRASOVSKⅡ N N, KOTEL'NIKOVA A N. Unification of differential games, generalized solutions of the Hamilton-Jacobi equations, and a stochastic guide[J]. Differential Equations, 2009, 45(11):1653-1668.
[5] KRIKELIS N, REKASIUS Z. On the solution of the optimal linear control problems under conflict of interest[J]. IEEE Transactions on Automatic Control, 1971, 16(2):140-147.
[6] INNOCENTI M, SCHMIDT D K. Quadratic optimal cooperative control synthesis with flight control application[J]. Journal of Guidance, Control, and Dynamics, 1984, 7(2):206-214.
[7] QIN C B, ZhANG H G, LUO Y H. Model-free adaptive dynamic programming for online optimal solution of the unknown nonlinear zero-sum differential game[C]//Proceedings of the 2014 International Joint Conference on Neural Networks. Piscataway:IEEE Press, 2014:3815-3820.
[8] LIU Y F, QI N, TANG Z W. Linear quadratic differential game strategies with two-pursuit versus single-evader[J]. Chinese Journal of Aeronautics, 2012, 25(6):896-905.
[9] 谢剑. 基于微分博弈论的多无人机追逃协同机动技术研究[D]. 哈尔滨:哈尔滨工业大学, 2015:32-45. XIE J. Differential game theory for multi UAV pursuit maneuver technology based on collaborative research[D]. Harbin:Harbin Institute of Technology, 2005:32-45(in Chinese)
[10] ISLER V, KANNAN S, KHANNA S, et al. Randomized pursuit-evasion in a polygonal environment[J]. IEEE Transactions on Robotics, 2005, 21(5):875-884.
[11] YAMAGUCHI H. A cooperative hunting behavior by mobile-robot troops[J]. The International Journal of Robotics Research, 2016, 18(9):931-940.
[12] CHEN J, ZHA W Z, PENG Z H, et al. Multi-player pursuit-evasion games with one superior evader[J]. Automatica, 2016, 71(71):24-32.
[13] PAN S, HUANG H, DING J, et al. Pursuit, evasion and defense in the plane[C]//Advances in Computing and Communications, 2012:4167-4173.
[14] FANG B F, PAN Q S, HONG B R, et al. Research on high speed evader vs multi lower speed pursuers in multi pursuit-evasion games[J]. Information Technology Journal, 2012, 11(8):989-997.
[15] CHEN X, WANG Y F. Study on multi-UAV air combat game based on fuzzy strategy[J]. Applied Mechanics and Materials, 2014, 494-495:1102-1105.
[16] WANG H P, YUE Q, LIU J T. Research on pursuit-evasion games with multiple heterogeneous pursuers and a high speed evader[C]//Chinese Control and Decision Conference. Piscataway:IEEE Press, 2015:4366-4370.
[17] AWHEDA M D, SCHWARTZ H M. A decentralized fuzzy learning algorithm for pursuit-evasion differential games with superior evaders[J]. Journal of Intelligent & Robotic Systems, 2016, 83(1):35-53.
[18] LU X S. Multi-agent reinforcement learning in games[D]. Ottawa:Carleton University, 2012:38-159.
[19] CHIDOZIE V A. Multi-robot learning in the guarding a territory game[D]. Ottawa:Carleton University, 2016:36-77.
[20] DUMAN E, KAYA M, AKIN E, et al. A multi-agent fuzzy-reinforcement learning method for continuous domains[C]//Multi Agent Systems and Applications IV, Lecture Notes in Computer Science. Berlin:Springer-Verlag, 2005:306-315.
[21] DAHL F A, HALCK O M. Minimax td-learning with neural nets in a Markov game[C]//European Conference on Machine Learning, 2000:117-128.
[22] BOWLING M, VELOSO M. Rational and convergent learning in stochastic games[C]//International Joint Conference on Artificial, 2001:1021-1026.
[23] BUSONIU L, BABUSKA R, SCHUTTER B D. Multi-agent reinforcement learning:An overview[M]. Berlin:Springer, 2010:1-3.
[24] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[EB/OL].[2020-04-20]. http://www.cs.utoronto.ca/~fidler/teaching/2015/slides/CSC2523/jonathan_rl.pdf.
[25] 左家亮, 杨任农, 张滢, 等. 基于启发式强化学习的空战机动智能决策[J]. 航空学报, 2017, 38(10):321168. ZUO J L, YANG R N, ZHANG Y, et al. Intelligent decision-making in air combat maneuvering based on heuristic reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2017, 38(10):321168(in Chinese)
[26] SILVER D, LEVER G, HEESS N, et al. Deterministic policy gradient algorithms[C]//International Conference on Machine Learning, 2014:387-395.

Cooperative attack-defense game of multiple UAVs with asymmetric maneuverability

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Honglin ZHANG, Jianjun LUO, Weihua MA. Spacecraft game decision making for threat avoidance of space targets based on machine learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(8): 329136-329136.
[2]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.
[3]	Shengzhe SHAN, Weiwei ZHANG. Air combat intelligent decision-making method based on self-play and deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328723-328723.
[4]	Jiaxiu YANG, Xinkai LI, Hongli ZHANG, Hao WANG. Time-varying formation control for heterogeneous clusters with switching topologies via reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(10): 329166-329166.
[5]	Bing XIAO, Haichao ZHANG. Reinforcement learning robust optimal control for spacecraft attitude stabilization [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(1): 628890-628890.
[6]	Xuejian WANG, Yongming WEN, Xiaorong SHI, Ningning ZHANG, Jiexi LIU. Design of hybrid intelligent decision framework for multi⁃agent and multi⁃coupling tasks [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729770-729770.
[7]	Youpeng DENG, Jiaxuan FAN, Yan ZHENG, Zhenya WANG, Yongliang LYU, Yuxiao LI. Multiagent opponent modeling with incompleted information [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729782-729782.
[8]	Weilin NI, Yonghai WANG, Cong XU, Fenghua CHI, Haizhao LIANG. Cooperative game guidance method for hypersonic vehicles based on reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729400-729400.
[9]	Zhilin FAN, Hongyong YANG, Yilin HAN. Target round-up control for multi-agent systems based on reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S1): 727487-727487.
[10]	Xiaowei FU, Zhe XU, Jindong ZHU, Nan WANG. Maneuvering decision-making of multi-UAV attack-defence confrontation based on PER-MATD3 [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(7): 327083-327083.
[11]	Xizhen GAO, Liang TANG, Huang HUANG. Deep reinforcement learning in autonomous manipulation for celestial bodies exploration: Applications and challenges [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(6): 26762-026762.
[12]	Pan ZHOU, Jiangtao HUANG, Sheng ZHANG, Gang LIU, Bowen SHU, Jigang TANG. Intelligent air combat decision making and simulation based on deep reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(4): 126731-126731.
[13]	Yupeng FU, Xiangyang DENG, Ziqiang ZHU, Limin ZHANG. Value-filter based air-combat maneuvering optimization [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(22): 628871-628871.
[14]	Chenglei YUE, Xuechuan WANG, Xiaokui YUE, Ting SONG. A spacecraft rendezvous and docking method based on inverse reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 328420-328420.
[15]	Xiangwei ZHU, Dan SHEN, Kai XIAO, Yuexin MA, Xiang LIAO, Fuqiang GU, Fangwen YU, Kefu GAO, Jingnan LIU. Mechanisms, algorithms, implementation and perspectives of brain⁃inspired navigation [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 28569-028569.