ACTA AERONAUTICAET ASTRONAUTICA SINICA >
A review of autonomous maneuver decision methods for unmanned combat aerial vehicle
Received date: 2024-06-27
Revised date: 2024-07-22
Accepted date: 2024-10-07
Online published: 2024-10-11
Supported by
National Natural Science Foundation of China(62101590)
Autonomous maneuver decision is a key technology in air-to-air confrontation, and the study of autonomous maneuver decision involves an optimal maneuver solution method. Through the study of autonomous maneuver decision method, the real-time and accuracy of autonomous maneuver decision of Unmanned Combat Aerial Vehicle(UCAV) in an aerial combat engagement can be improved, which has important theoretical research significance and application value in the promotion of UCAV’s autonomous aerial combat and manned/unmanned aircraft cooperative aerial combat. Currently, a large number of studies have been conducted around the theories of mathematical solution, data-driven, intelligent optimization and their applications, which have given a greater impetus to the research of autonomous maneuver decision methods and their applications. Firstly, the basic concept of autonomous maneuver decision of UCAV is elaborated, then the research progress of maneuver decision methods is reviewed, several methods commonly used in maneuver decision research are introduced, the maneuver decision methods are classified and summarized and the performance of several typical maneuver decision methods in air combat simulation is compared. Finally, the difficulties and prospects of autonomous maneuver decision research are pointed out.
Yuequn LUO , Dali DING , Mulai TAN , Yidong LIU , Huan ZHOU . A review of autonomous maneuver decision methods for unmanned combat aerial vehicle[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2025 , 46(7) : 30877 -030877 . DOI: 10.7527/S1000-6893.2024.30877
1 | 孙智孝, 杨晟琦, 朴海音, 等. 未来智能空战发展综述[J]. 航空学报, 2021, 42(8): 525799. |
SUN Z X, YANG S Q, PIAO H Y, et al. A survey of air combat artificial intelligence[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(8): 525799 (in Chinese). | |
2 | 陈浩, 黄健, 刘权, 等. 自主空战机动决策技术研究进展与展望[J]. 控制理论与应用, 2023, 40(12): 2104-2129. |
CHEN H, HUANG J, LIU Q, et al. Review and prospects of autonomous air combat maneuver decisions[J].Journal of Control Theory and Applications, 2023, 40(12): 2104-2129 (in Chinese). | |
3 | DUAN H B, LEI Y Q, XIA J, et al. Autonomous maneuver decision for unmanned aerial vehicle via improved pigeon-inspired optimization?[J]. IEEE Transactions on Aerospace and Electronic Systems, 2023, 59(3): 3156-3170. |
4 | LIU Y F, QI N M, TANG Z W. Linear quadratic differential game strategies with two-pursuit versus single-evader[J]. Chinese Journal of Aeronautics, 2012, 25(6): 896-905. |
5 | PARK H, LEE B Y, TAHK M J, et al. Differential game based air combat maneuver generation using scoring function matrix[J]. International Journal of Aeronautical and Space Sciences, 2016, 17(2): 204-213. |
6 | WANG M L, WANG L X, YUE T, et al. Influence of unmanned combat aerial vehicle agility on short-range aerial combat effectiveness?[J]. Aerospace Science and Technology, 2020, 96: 105534. |
7 | LóPEZ N R, ?BIKOWSKI R. Effectiveness of autonomous decision making for unmanned combat aerial vehicles in dogfight engagements[J]. Journal of Guidance,Control, and Dynamics, 2018, 41(4): 1021-1024. |
8 | LI S Y, CHEN M, WANG Y H, et al. Air combat decision-making of multiple UCAVs based on constraint strategy games[J]. Defence Technology, 2022, 18(3): 368-383. |
9 | VIRTANEN K, RAIVIO T, HAMALAINEN R P. Modeling pilot’s sequential maneuvering decisions by a multistage influence diagram[J]. Journal of Guidance, Control, and Dynamics, 2004, 27(4): 665-677. |
10 | EKLUND J M, SPRINKLE J, SASTRY S S. Switched and symmetric pursuit/evasion games using online model predictive control with application to autonomous aircraft[J]. IEEE Transactions on Control Systems Technology, 2012, 20(3): 604-620. |
11 | HUANG C Q, DONG K S, HUANG H Q, et al. Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization[J]. Journal of Systems Engineering and Electronics, 2018, 29(1): 86-97. |
12 | MCGREW J S, HOW J P, WILLIAMS B, et al. Air-combat strategy using approximate dynamic programming[J]. Journal of Guidance, Control, and Dynamics, 2010, 33(5): 1641-1654. |
13 | ZHANG H P, HUANG C Q. Maneuver decision-making of deep learning for UCAV thorough azimuth angles[J]. IEEE Access, 2020, 8: 12976-12987. |
14 | YANG Q M, ZHANG J D, SHI G Q, et al. Maneuver decision of UAV in short-range air combat based on deep reinforcement learning?[J]. IEEE Access, 2020, 8: 363-378. |
15 | ERNEST N, CARROLL D, SCHUMACHER C, et al.Genetic fuzzy based artificial intelligence for unmanned combat aerial vehicle control in simulated air combat missions?[J]. Journal of Defense Management, 2016, 6(1): 144. |
16 | DUAN H B, LI P, YU Y X. A predator-prey particle swarm optimization approach to multiple UCAV air combat modeled by dynamic game theory?[J]. IEEE/CAA Journal of Automatica Sinica, 2015, 2(1): 11-18. |
17 | HAN T, WANG X F, LIANG Y J, et al. Study on UCAV robust maneuvering decision in automatic air combat based on target accessible domain[J]. Journal of Physics: Conference Series, 2019, 1213(5): 052004. |
18 | HO E, RAJAGOPALAN A, SKVORTSOV A, et al. Game theory in defence applications: A review[J]. Sensors, 2022, 22(3): 1032. |
19 | 钟友武, 杨凌宇, 柳嘉润, 等. 基于智能微分对策的自主机动决策方法研究[J]. 飞行力学, 2008, 26(6): 29-33. |
ZHONG Y W, YANG L Y, LIU J R, et al. Method of autonomous maneuver decision based on intelligent differential game[J]. Flight Dynamics,2008, 26(6): 29-33 (in Chinese). | |
20 | 王义宁, 姜玉宪. 空战决策中的智能微分对策法[J]. 飞行力学, 2003, 21(1): 66-70. |
WANG Y N, JIANG Y X. An intelligent differential gameon air combat decision[J]. Flight Dynamics, 2003, 21(1): 66-70 (in Chinese). | |
21 | LEE B Y, HAN S, PARK H J, et al. One-versus-one air-to-air combat maneuver generation based on the differential game[C]∥ Proceedings of the 2016 Congress of the International Council of the Aeronautical Sciences. ICAS, 2016: 1-7. |
22 | HORIE K, CONWAY B A. Optimal fighter pursuit-evasion maneuvers found via two-sided optimization[J]. Journal of Guidance, Control, and Dynamics, 2006, 29(1): 105-112. |
23 | XU G Y, WEI S N, ZHANG H M. Application of situation function in air combat differential games[C]∥ 2017 36th Chinese Control Conference (CCC). Piscataway: IEEE Press, 2017: 5865-5870. |
24 | GARCIA E, MOLLA V, CASBEER D W, et al. Strategies for defending a coastline against multiple attackers[C]∥ 2019 IEEE 58th Conference on Decision and Control (CDC). Piscataway: IEEE Press, 2019: 7319-7324. |
25 | VIRTANEN K, KARELAHTI J, RAIVIO T. Modeling air combat by a moving horizon influence diagram game[J]. Journal of Guidance, Control, and Dynamics, 2006, 29(5): 1080-1091. |
26 | ZHONG L, TONG M A, ZHONG W, et al. Sequential maneuvering decisions based on multi-stage influence diagram in air combat[J]. Journal of Systems Engineering and Electronics, 2007, 18(3): 551-555. |
27 | LU H C, WU B Y, CHEN J Q. Fighter equipment contribution evaluation based on maneuver decision?[J]. IEEE Access, 2021, 9: 132241-132254. |
28 | PAN Q, ZHOU D Y, HUANG J C, et al. Maneuver decision for cooperative close-range air combat based on state predicted influence diagram[C]∥ 2017 IEEE International Conference on Information and Automation (ICIA). Piscataway: IEEE Press, 2017: 726-731. |
29 | ZHOU S Y, WANG Z J, FAN G, et al. Collaborative maneuvering decision based on multi-layer influence diagram group decision-making[M]∥ Advances in Transdisciplinary Engineering. Netherlands: IOS Press, 2024:345-357. |
30 | 钟麟, 佟明安, 钟卫. 影响图对策在多机协同空战中的应用[J]. 北京航空航天大学学报, 2007, 33(4): 450-453. |
ZHONG L, TONG M A, ZHONG W. Application of multistage influence diagram game theory for multiple cooperative air combat[J]. Journal of Beijing University of Aeronautics and Astronautics, 2007, 33(4): 450-453 (in Chinese). | |
31 | GENG W X, KONG F E, MA D Q. Study on tactical decision of UAV medium-range air combat[C]∥ The 26th Chinese Control and Decision Conference (2014 CCDC). Piscataway: IEEE Press, 2014: 135-139. |
32 | WANG X F, WANG B S. Situation assessment method based on Bayesian network and intuitionistic fuzzy reasoning[J]. System Engineering and Electronics, 2009, 31(11): 2742-2746. |
33 | 刘守业. 非完备信息下无人机空战决策与导引方法研究[D]. 沈阳: 沈阳航空航天大学, 2022: 3-28. |
LIU S Y. Decision-making and guidance method of unmanned aerial vehicle under incomplete information[D]. Shenyang: Shenyang Aerospace University, 2022: 3-28 (in Chinese). | |
34 | PIAO H Y, SUN Z X, MENG G L, et al. Beyond-visual-range air combat tactics auto-generation by reinforcement learning[C]∥ 2020 International Joint Conference on Neural Networks (IJCNN). Piscataway: IEEE Press, 2020: 1-8. |
35 | 张强, 杨任农, 俞利新, 等. 基于Q-network强化学习的超视距空战机动决策[J]. 空军工程大学学报, 2018, 19(6): 8-14. |
ZHANG Q, YANG R N, YU L X, et al. BVR air combat maneuvering decision by using Q-network reinforcement learning[J]. Journal of Air Force Engineering University, 2018, 19(6): 8-14 (in Chinese). | |
36 | YANG Q M, ZHU Y, ZHANG J D, et al. UAV air combat autonomous maneuver decision based on DDPG algorithm[C]?∥ 2019 IEEE 15th International Conference on Control and Automation (ICCA). Piscataway: IEEE Press, 2019: 37-42. |
37 | KONG W R, ZHOU D Y, YANG Z, et al. UAV autonomous aerial combat maneuver strategy generation with observation error based on state-adversarial deep deterministic policy gradient and inverse reinforcement learning[J]. Electronics, 2020, 9(7): 1121. |
38 | 黄长强, 赵克新, 韩邦杰, 等. 一种近似动态规划的无人机机动决策方法[J]. 电子与信息学报, 2018, 40(10): 2447-2452. |
HUANG C Q, ZHAO K X, HAN B J, et al. Maneuvering decision-making method of UAV based on approximate dynamic programming[J]. Journal of Electronics & Information Technology, 2018, 40(10): 2447-2452 (in Chinese). | |
39 | MA Y F, MA X L, SONG X. A case study on air combat decision using approximated dynamic programming[J]. Mathematical Problems in Engineering, 2014(4): 183401. |
40 | 梅丹, 刘锦涛, 高丽. 基于近似动态规划与零和博弈的空战机动决策[J]. 兵工自动化, 2017, 36(3): 35-39. |
MEI D, LIU J T, GAO L. Maneuver decision of air combat based on approximate dynamic programming and zero-sum game?[J]. Ordnance Industry Automation, 2017, 36(3): 35-39 (in Chinese). | |
41 | HU Z C, GAO P, WANG F. Research on autonomous maneuvering decision of UCAV based on approximate dynamic programming[C]∥ 2019 International Conference on Image and Video Processing, and Artificial Intelligence. NewYork: SPIE, 2019: 636-641. |
42 | 姜龙亭, 寇雅楠, 王栋, 等. 改进近似动态规划法的攻击占位决策[J]. 火力与指挥控制, 2019, 44(7): 135-141. |
JIANG L T, KOU Y N, WANG D, et al. Attack placeholder decision based on improved approximate dynamic programming[J]. Fire Control & Command Control, 2019, 44(7): 135-141 (in Chinese). | |
43 | CRUMPACKER J B, ROBBINS M J, JENKINS P R. An approximate dynamic programming approach for solving an air combat maneuvering problem[J]. Expert Systems with Applications, 2022, 203: 117448. |
44 | ARULKUMARAN K, DEISENROTH M P, BRUNDAGE M, et al. Deep reinforcement learning: A brief survey?[J]. IEEE Signal Processing Magazine, 2017, 34(6): 26-38. |
45 | SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016, 529: 484-489. |
46 | SILVER D, SCHRITTWIESER J, SIMONYAN K, et al. Mastering the game of Go without human knowledge[J]. Nature, 2017, 550: 354-359. |
47 | VINYALS O, BABUSCHKIN I, CZARNECKI W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. Nature, 2019, 575: 350-354. |
48 | BERNER C, BROCKMAN G, CHAN B, et al. Dota 2 with large scale deep reinforcement learning[DB/OL]. arXiv preprint: 1912.06680, 2019. |
49 | NGUYEN T T, NGUYEN N D, NAHAVANDI S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications?[J]. IEEE Transactions on Cybernetics, 2020, 50(9): 3826-3839. |
50 | LYU L, SHEN Y, ZHANG S C. The advance of reinforcement learning and deep reinforcement learning[C]∥ 2022 IEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA). Piscataway: IEEE Press, 2022: 644-648. |
51 | SARKAR N I, GUL S. Artificial intelligence-based autonomous UAV networks: A survey[J]. Drones, 2023, 7(5): 322. |
52 | BAYERLEIN H, THEILE M, CACCAMO M, et al. Multi-UAV path planning for wireless data harvesting with deep reinforcement learning[J]. IEEE Open Journal of the Communications Society, 2021, 2: 1171-1187. |
53 | ZHU B T, BEDEER E, NGUYEN H H, et al. UAV trajectory planning in wireless sensor networks for energy consumption minimization by deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2021, 70(9): 9540-9554. |
54 | WANG X, CHEN Y D, ZHU W W. A survey on curriculum learning?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4555-4576. |
55 | WANG Y B, JIANG T S, LI Y J. A hierarchical reinforcement learning method on multi UCAV air combat[C]?∥ 2021 International Conference on Neural Networks, Information and Communication Engineering. New York: SPIE, 2021, 11933: 117-123. |
56 | WANG L, WANG K Z, PAN C H, et al. Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing[J]. IEEE Transactions on Cognitive Communications and Networking, 2021, 7(1): 73-84. |
57 | ZHAO W L, MENG Z J, WANG K P, et al. Hierarchical active tracking control for UAVs via deep reinforcement learning?[J]. Applied Sciences, 2021, 11(22): 10595. |
58 | LI B H, WU Y J. Path planning for UAV ground target tracking via deep reinforcement learning?[J]. IEEE Access, 2020, 8: 29064-29074. |
59 | LI B, GAN Z G, CHEN D Q, et al. UAV maneuvering target tracking in uncertain environments based on deep reinforcement learning and meta-learning?[J]. Remote Sensing, 2020, 12(22): 3789. |
60 | LI Y, HAN W, WANG Y Q. Deep reinforcement learning with application to air confrontation intelligent decision-making of manned/unmanned aerial vehicle cooperative system?[J]. IEEE Access, 2020, 8: 67887-67898. |
61 | WU L Z, WANG C, ZHANG P P, et al. Deep reinforcement learning with corrective feedback for autonomous UAV landing on a mobile platform[J]. Drones, 2022, 6(9): 238. |
62 | XIE J Y, PENG X D, WANG H J, et al. UAV autonomous tracking and landing based on deep reinforcement learning strategy[J]. Sensors, 2020, 20(19): 5630. |
63 | XU D, GUO Y X, YU Z Y, et al. PPO-Exp: Keeping fixed-wing UAV formation with deep reinforcement learning[J]. Drones, 2022, 7(1): 28. |
64 | ZHAO W W, CHU H R, MIAO X K, et al. Research on the multiagent joint proximal policy optimization algorithm controlling cooperative fixed-wing UAV obstacle avoidance[J]. Sensors, 2020, 20(16): 4546. |
65 | ZHAO Y, CHEN Y F, ZHEN Z Y, et al. Multi-weapon multi-target assignment based on hybrid genetic algorithm in uncertain environment[J]. International Journal of Advanced Robotic Systems, 2020, 17(2): 1729881420905922. |
66 | ZHAO X R, YANG R N, ZHANG Y, et al. Deep reinforcement learning for intelligent dual-UAV reconnaissance mission planning[J]. Electronics, 2022, 11(13): 2031. |
67 | YUE L F, YANG R N, ZHANG Y, et al. Deep reinforcement learning for UAV intelligent mission planning[J]. Complexity, 2022(1): 3551508. |
68 | GUO T, JIANG N, LI B Y, et al. UAV navigation in high dynamic environments: A deep reinforcement learning approach[J]. Chinese Journal of Aeronautics, 2021, 34(2): 479-489. |
69 | HU J W, WANG L H, HU T M, et al. Autonomous maneuver decision making of dual-UAV cooperative air combat based on deep reinforcement learning[J]. Electronics, 2022, 11(3): 467. |
70 | WANG X W, WANG Y H, SU X C, et al. Deep reinforcement learning-based air combat maneuver decision-making: Literature review, implementation tutorial and future direction[J]. Artificial Intelligence Review, 2023, 57(1): 1. |
71 | CAO Y, KOU Y X, LI Z W, et al. Autonomous maneuver decision of UCAV air combat based on double deep Q network algorithm and stochastic game theory[J]. International Journal of Aerospace Engineering, 2023(1): 3657814. |
72 | ZHANG X B, LIU G Q, YANG C J, et al. Research on air confrontation maneuver decision-making method based on reinforcement learning[J]. Electronics, 2018, 7(11): 279. |
73 | PIAO H Y, HAN Y, CHEN H C, et al. Complex relationship graph abstraction for autonomous air combat collaboration: A learning and expert knowledge hybrid approach?[J]. Expert Systems with Applications, 2023, 215: 119285. |
74 | SUN Z X, PIAO H Y, YANG Z, et al. Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play[J]. Engineering Applications of Artificial Intelligence, 2021, 98: 104112. |
75 | LI Y F, SHI J P, JIANG W, et al. Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MS-DDQN algorithm[J]. Defence Technology, 2022, 18(9): 1697-1714. |
76 | ZHANG H P, ZHOU H, WEI Y J, et al. Autonomous maneuver decision-making method based on reinforcement learning and Monte Carlo tree search[J]. Frontiers in Neurorobotics, 2022, 16: 996412. |
77 | YANG K B, DONG W H, CAI M, et al. UCAV air combat maneuver decisions based on a proximal policy optimization algorithm with situation reward shaping[J]. Electronics, 2022, 11(16): 2602. |
78 | ZHENG Z Q, DUAN H B. UAV maneuver decision-making via deep reinforcement learning for short-range air combat[J]. Intelligence & Robotics, 2023, 3(1): 76-94. |
79 | 杨书恒, 张栋, 熊威, 等. 基于可解释性强化学习的空战机动决策方法[J]. 航空学报, 2024, 45(18): 329922. |
YANG S H, ZHANG D, XIONG W, et al. Decision-making method for air combat maneuver based on explainable reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(18): 329922 (in Chinese). | |
80 | 谢建峰, 杨啟明, 戴树岭, 等. 基于强化遗传算法的无人机空战机动决策研究[J]. 西北工业大学学报, 2020, 38(6): 1330-1338. |
XIE J F, YANG Q M, DAI S L, et al. Air combat maneuver decision based on reinforcement genetic algorithm[J]. Journal of Northwestern Polytechnical University, 2020, 38(6): 1330-1338 (in Chinese). | |
81 | 周德云, 李锋, 蒲小勃, 等. 基于遗传算法的飞机战术飞行动作决策[J]. 西北工业大学学报, 2002, 20(1): 109-112. |
ZHOU D Y, LI F, PU X B, et al. On improving tactical planning in air combat in P.R.China with genetic algorithm[J]. Journal of Northwestern Polytechnical University, 2002, 20(1): 109-112 (in Chinese). | |
82 | 王杰, 丁达理, 董康生, 等. UCAV自主空战战术机动动作建模与轨迹生成[J]. 火力与指挥控制, 2018, 43(12): 42-49. |
WANG J, DING D L, DONG K S, et al. UCAV autonomous air combat tactical maneuvering modeling and trajectory generation[J]. Fire Control & Command Control, 2018, 43(12): 42-49 (in Chinese). | |
83 | 张涛, 于雷, 周中良, 等. 基于变权重伪并行遗传算法的空战机动决策[J]. 飞行力学, 2012, 30(5): 470-474. |
ZHANG T, YU L, ZHOU Z L, et al. Decision-making for aircombat maneuvering based on variable weight pseudo-parallel genetical algorithm?[J]. Flight Dynamics, 2012, 30(5): 470-474 (in Chinese). | |
84 | BURGIN G H, SIDOR L B. Rule-based air combat simulation: NASA-CR-4160[R]. Washington,D.?C.: National Aeronautics and Space Administration, 1988. |
85 | 王锐平, 高正红. 无人机空战仿真中基于机动动作库的决策模型[J]. 飞行力学, 2009, 27(6): 72-75, 79. |
WANG R P, GAO Z H. Research on decision system in air combat simulation using maneuver library[J]. Flight Dynamics, 2009, 27(6): 72-75, 79 (in Chinese). | |
86 | 王刚, 雷英杰, 何晶. 空战决策指挥引导专家系统[J]. 空军工程大学学报(自然科学版), 2002, 3(1): 11-13. |
WANG G, LEI Y J, HE J. Interception guidance expert system for airfight decision[J]. Journal of Air Force Engineering University (Natural Science Edition), 2002, 3(1): 11-13 (in Chinese). | |
87 | 谭目来, 丁达理, 谢磊, 等. 基于模糊专家系统与IDE算法的UCAV逃逸机动决策[J]. 系统工程与电子技术, 2022, 44(6): 1984-1993. |
TAN M L, DING D L, XIE L, et al. UCAV escape maneuvering decision based on fuzzy expert system and IDE algorithm?[J]. Systems Engineering and Electronics, 2022, 44(6): 1984-1993 (in Chinese). | |
88 | QIAN C X, ZHANG X B, LI L, et al. H3E: Learning air combat with a three-level hierarchical framework embedding expert knowledge[J]. Expert Systems with Applications, 2024, 245: 123084. |
89 | CHAI R Q, TSOURDOS A, SAVVARIS A, et al. Solving constrained trajectory planning problems using biased particle swarm optimization[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021, 57(3): 1685-1701. |
90 | RUAN W Y, DUAN H B, DENG Y M. Autonomous maneuver decisions via transfer learning pigeon-inspired optimization for UCAVs in dogfight engagements[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(9): 1639-1657. |
91 | TAN M L, TANG A D, DING D L, et al. Autonomous air combat maneuvering decision method of UCAV based on LSHADE-TSO-MPC under enemy trajectory prediction[J]. Electronics, 2022, 11(20): 3383. |
/
〈 |
|
〉 |