基于DDPG算法的无人机集群追击任务

doi:10.7527/S1000-6893.2020.24000

电子电气工程与控制

本期目录 | 过刊浏览 | 高级检索

前一篇 | 后一篇

基于DDPG算法的无人机集群追击任务

张耀中¹, 许佳林¹, 姚康佳¹, 刘洁凌²

1. 西北工业大学电子信息学院, 西安 710072;
2. 西安北方光电科技防务有限公司, 西安 710043

收稿日期:2020-03-21 修回日期:2020-06-15 发布日期:2020-06-12
通讯作者: 张耀中 E-mail:zhang_y_z@nwpu.edu.cn
基金资助:
航空科学基金（2017ZC53033）

Pursuit missions for UAV swarms based on DDPG algorithm

ZHANG Yaozhong¹, XU Jialin¹, YAO Kangjia¹, LIU Jieling²

1. School of Electronics and Information, Northwestern Polytechnical University, Xi'an 710072, China;
2. Xi'an North Electro-optic Science & Technology Co. Ltd, Xi'an 710043, China

Received:2020-03-21 Revised:2020-06-15 Published:2020-06-12
Supported by:
Aeronautical Science Foundation of China (2017ZC53033)

摘要/Abstract

摘要： 无人机的集群化应用技术是近年来的研究热点，随着无人机自主智能的不断提高，无人机集群技术必将成为未来无人机发展的主要趋势之一。针对无人机集群协同执行对敌方来袭目标的追击任务，构建了典型的任务场景，基于深度确定性策略梯度网络（DDPG）算法，设计了一种引导型回报函数有效解决了深度强化学习在长周期任务下的稀疏回报问题，通过引入基于滑动平均值的软更新策略减少了DDPG算法中Eval网络和Target网络在训练过程中的参数震荡，提高了算法的训练效率。仿真结果表明，训练完成后的无人机集群能够较好地执行对敌方来袭目标的追击任务，任务成功率达到95%。可以说无人机集群技术作为一种全新概念的作战模式在军事领域具有潜在的应用价值，人工智能算法在无人机集群的自主决策智能化发展方向上具有一定的应用前景。

关键词: DDPG算法, 无人机集群, 任务决策, 深度强化学习, 稀疏回报

Abstract: The Unmanned Aerial Vehicle (UAV) swarm technology is one of the research hotspots in recent years. With continuous advancement in autonomous intelligence of UAVs, the UAV swarm technology is bound to become one of the main trends of UAV development in the future. In view of the collaborative pursuit missions of UAV swarms against the enemy, we establish a typical task scenario, and, based on the Deep Deterministic Policy Gradient (DDPG) algorithm, design a guided reward function which effectively solves the sparse rewards problem of deep intensive learning during long-period missions. We introduce a sliding average based soft updating strategy to reduce parameter oscillations in the Eval network and the target network during the training process, thereby improving the training efficiency. The simulation results show that after training, the UAV swarm can successfully carry out the pursuit missions with a success rate of 95%. The UAV swarm technology as a brand new combat mode has a potential application value for application in the military field, and this artificial intelligence algorithm has a certain application prospect in the development of autonomous decision-making by UAV swarms.

Key words: DDPG algorithm, UAV swarms, task decision, deep reinforcement learning, sparse rewards

中图分类号:

V279

张耀中, 许佳林, 姚康佳, 刘洁凌. 基于DDPG算法的无人机集群追击任务[J]. 航空学报, 2020, 41(10): 324000-324000.

ZHANG Yaozhong, XU Jialin, YAO Kangjia, LIU Jieling. Pursuit missions for UAV swarms based on DDPG algorithm[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(10): 324000-324000.

参考文献

[1] PHAM H X, LA H M, FEILSEIFER D, et al. Autonomous UAV navigation using reinforcement learning[EB/OL]. (2018-01-16)[2020-03-10]. https://arxiv.org/abs/1801.05086.
[2] PHAM H X, LA H M, FEILSEIFER D, et al. Cooperative and distributed reinforcement learning of drones for field coverage[J].(2018-09-16)[2020-03-10]. https://arxiv.org/abs/1803.07250.
[3] QI S, ZHU S. Intent-aware multi-agent reinforcement learning[C]//2018 IEEE International Conference on Robotics and Automation (ICRA). Piscataway:IEEE Press, 2018:7533-7540.
[4] 李高垒, 马耀飞. 基于深度网络的空战态势特征提取[J].系统仿真学报, 2017, 29(S1):98-105, 112. LI G L, MA Y F. Feature extraction algorithm of air combat situation based on deep neural networks[J].Journal of System Simulation, 2017, 29(S1):98-105, 112(in Chinese).
[5] 魏航. 基于强化学习的无人机空中格斗算法研究[D]. 哈尔滨:哈尔滨工业大学, 2015. WEI H. Resarch of UCAV air combat based on reinforcement learning[D]. Harbin:Harbin Institute of Technology,2015(in Chinese).
[6] YAMAGUCHI H. A cooperative hunting behavior by mobile robot troops[C]//Proceedings 1998 IEEE International Conference on Robotics and Automation. Piscataway:IEEE Press, 1998:931-940.
[7] GADRE A. Learning strategies in multi-agent systems applications to the herding problem[D]. Blacksburg:Virginia Polytechnic Institute and State University, 2001.
[8] 苏治宝, 陆际联, 童亮. 一种多移动机器人协作围捕策略[J].北京理工大学学报, 2004(5):32-35, 44. SU Z B, LU J L, TONG L. Strategy of cooperative hunting by multiple mobile robots[J].Beijing Institute of Technology, 2004(5):32-35, 44(in Chinese).
[9] 罗德林, 徐扬, 张金鹏. 无人机集群对抗技术新进展[J].科技导报, 2017,35(7):26-31. LUO D L, XU Y, ZHANG J P. New progresses on UAV swarm confrontation[J].Science & Technology Review, 2017,35(7):26-31(in Chinese).
[10] CARL E J. Analysis of fatigue, fatigue-crack propagation and fracture data:AIAA-2009-1363[R]. Reston:AIAA, 2009.
[11] ZUHAIR Q M, SONGHAO P, HAIYANG J, et al. A novel approach for multi-agent cooperative pursuit to capture grouped evaders[J].The Journal of Supercomputing, 2018, 76:3416-3426.
[12] ZHAOYI P, SONGHAO P, MOHAMMED E H S, et al. Coalition formation for multi-agent pursuit based on neural network[J].Journal of Intelligent & Robotic Systems, 2019, 95(1):887-899.
[13] HUMAYOO M, CHENG X. Relative importance sampling for off-policy actor-critic in deep reinforcement learning[EB/OL]. (2019-07-19)[2020-03-10]. https://arxiv.org/abs/1810.12558?context=cs.
[14] 刘建伟, 高峰, 罗雄麟. 基于值函数和策略梯度的深度强化学习综述[J].计算机学报, 2019, 42(6):1406-1438. LIU J W, GAO F, LUO X L. A survey of deep reinforcement learning based on value function and strategy gradient[J].Chinese Journal of Computers, 2019, 42(6):1406-1438(in Chinese).
[15] WANG G, SHI J. Actor-critic for multi-agent system with variable quantity of agents[C]//International Conference on Internet of Things as a Service, 2017:48-56.
[16] HUANG W, WANG Y, YI X. A deep reinforcement learning approach to preserve connectivity for multi-robot systems[C]//2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). Piscataway:IEEE Press, 2017:1-7.
[17] YI H. Deep deterministic policy gradient for autonomous vehicle driving[C]//Proceedings on the International Conference on Artificial Intelligence (ICAI), 2018:191-194.
[18] ANDERSEN P, GOODWIN M, GRANMO O. Deep RTS:A game environment for deep reinforcement learning in real-time strategy games[C]//2018 IEEE Conference on Computational Intelligence and Games (CIG). Piscataway:IEEE Press, 2018:1-8.
[19] DILOKTHANAKUL N, KAPLANIS C, PAWLOWSKI N, et al. Feature control as intrinsic motivation for hierarchical reinforcement learning[J].IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(11):3409-3418.
[20] NIE H, CHEN Y, SONG Y, et al. A general real-time OPF algorithm using DDPG with multiple simulation platforms[C]//2019 IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia). Piscataway:IEEE Press, 2019:3713-3718.
[21] YANG Q, ZHU Y, ZHANG J, et al. UAV air combat autonomous maneuver decision based on DDPG algorithm[C]//2019 IEEE 15th International Conference on Control and Automation (ICCA). Piscataway:IEEE Press, 2019:37-42.
[22] BANERJEE A, GHOSH D, DAAS S. Evolving network topology in policy gradient reinforcement learning algorithms[C]//2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP), 2019:1-5.
[23] SHI H, SUN Y, LI G. Model-based DDPG for motor control[C]//2017 International Conference on Progress in Informatics and Computing (PIC), 2017:284-288.

E-mail：hkxb@buaa.edu.cn

关于我们

期刊社服务

专业学科

封面文章

友情链接

主管单位：中国科学技术协会主办单位：中国航空学会北京航空航天大学

基于DDPG算法的无人机集群追击任务

Pursuit missions for UAV swarms based on DDPG algorithm

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 11

编辑推荐

Metrics

本文评价

[1]	刘雷, 刘大卫, 王晓光, 陈俊男, 刘东兴. 无人机集群与反无人机集群发展现状及展望[J]. 航空学报, 2022, 43(S1): 726908-726908.
[2]	苏翎菲, 化永朝, 董希旺, 任章. 人与无人机集群多模态智能交互方法[J]. 航空学报, 2022, 43(S1): 727001-727001.
[3]	徐广通, 王祝, 曹严, 孙景亮, 龙腾. 动态优先级解耦的无人机集群轨迹分布式序列凸规划[J]. 航空学报, 2022, 43(2): 325059-325059.
[4]	任峰, 高传强, 唐辉. 机器学习在流动控制领域的应用及发展趋势[J]. 航空学报, 2021, 42(4): 524686-524686.
[5]	相晓嘉, 闫超, 王菖, 尹栋. 基于深度强化学习的固定翼无人机编队协调控制方法[J]. 航空学报, 2021, 42(4): 524009-524009.
[6]	唐帅文, 周志杰, 姜江, 曹友, 陈媛, 叶燕清. 考虑扰动的无人机集群协同态势感知一致性评估[J]. 航空学报, 2020, 41(S2): 724233-724233.
[7]	姜龙亭, 魏瑞轩, 张启瑞, 王栋. 基于群智机理的集群防碰撞控制[J]. 航空学报, 2020, 41(S2): 724294-724294.
[8]	徐磊, 周藜莎, 李仁俊, 顾村锋. 毫米波波束编码技术在无人机智能集群中的应用[J]. 航空学报, 2020, 41(S1): 723754-723754.
[9]	刘冰雁, 叶雄兵, 周赤非, 刘必鎏. 基于改进DQN的复合模式在轨服务资源分配[J]. 航空学报, 2020, 41(5): 323630-323630.
[10]	王祥科, 刘志宏, 丛一睿, 李杰, 陈浩. 小型固定翼无人机集群综述和未来发展综述[J]. 航空学报, 2020, 41(4): 23732-023732.
[11]	刘冰雁, 叶雄兵, 高勇, 王新波, 倪蕾. 基于分支深度强化学习的非合作目标追逃博弈策略求解[J]. 航空学报, 2020, 41(10): 324040-324040.