Reviews

A review of autonomous maneuver decision methods for unmanned combat aerial vehicle

  • Yuequn LUO ,
  • Dali DING ,
  • Mulai TAN ,
  • Yidong LIU ,
  • Huan ZHOU
Expand
  • 1.Graduate School,Air Force Engineering University,Xi’an 710038,China
    2.Aviation Engineering School,Air Force Engineering University,Xi’an 710038,China

Received date: 2024-06-27

  Revised date: 2024-07-22

  Accepted date: 2024-10-07

  Online published: 2024-10-11

Supported by

National Natural Science Foundation of China(62101590)

Abstract

Autonomous maneuver decision is a key technology in air-to-air confrontation, and the study of autonomous maneuver decision involves an optimal maneuver solution method. Through the study of autonomous maneuver decision method, the real-time and accuracy of autonomous maneuver decision of Unmanned Combat Aerial Vehicle(UCAV) in an aerial combat engagement can be improved, which has important theoretical research significance and application value in the promotion of UCAV’s autonomous aerial combat and manned/unmanned aircraft cooperative aerial combat. Currently, a large number of studies have been conducted around the theories of mathematical solution, data-driven, intelligent optimization and their applications, which have given a greater impetus to the research of autonomous maneuver decision methods and their applications. Firstly, the basic concept of autonomous maneuver decision of UCAV is elaborated, then the research progress of maneuver decision methods is reviewed, several methods commonly used in maneuver decision research are introduced, the maneuver decision methods are classified and summarized and the performance of several typical maneuver decision methods in air combat simulation is compared. Finally, the difficulties and prospects of autonomous maneuver decision research are pointed out.

Cite this article

Yuequn LUO , Dali DING , Mulai TAN , Yidong LIU , Huan ZHOU . A review of autonomous maneuver decision methods for unmanned combat aerial vehicle[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2025 , 46(7) : 30877 -030877 . DOI: 10.7527/S1000-6893.2024.30877

References

1 孙智孝, 杨晟琦, 朴海音, 等. 未来智能空战发展综述[J]. 航空学报202142(8): 525799.
  SUN Z X, YANG S Q, PIAO H Y, et al. A survey of air combat artificial intelligence[J]. Acta Aeronautica et Astronautica Sinica202142(8): 525799 (in Chinese).
2 陈浩, 黄健, 刘权, 等. 自主空战机动决策技术研究进展与展望[J]. 控制理论与应用202340(12): 2104-2129.
  CHEN H, HUANG J, LIU Q, et al. Review and prospects of autonomous air combat maneuver decisions[J].Journal of Control Theory and Applications202340(12): 2104-2129 (in Chinese).
3 DUAN H B, LEI Y Q, XIA J, et al. Autonomous maneuver decision for unmanned aerial vehicle via improved pigeon-inspired optimization?[J]. IEEE Transactions on Aerospace and Electronic Systems202359(3): 3156-3170.
4 LIU Y F, QI N M, TANG Z W. Linear quadratic differential game strategies with two-pursuit versus single-evader[J]. Chinese Journal of Aeronautics201225(6): 896-905.
5 PARK H, LEE B Y, TAHK M J, et al. Differential game based air combat maneuver generation using scoring function matrix[J]. International Journal of Aeronautical and Space Sciences201617(2): 204-213.
6 WANG M L, WANG L X, YUE T, et al. Influence of unmanned combat aerial vehicle agility on short-range aerial combat effectiveness?[J]. Aerospace Science and Technology202096: 105534.
7 LóPEZ N R, ?BIKOWSKI R. Effectiveness of autonomous decision making for unmanned combat aerial vehicles in dogfight engagements[J]. Journal of Guidance,Control, and Dynamics201841(4): 1021-1024.
8 LI S Y, CHEN M, WANG Y H, et al. Air combat decision-making of multiple UCAVs based on constraint strategy games[J]. Defence Technology202218(3): 368-383.
9 VIRTANEN K, RAIVIO T, HAMALAINEN R P. Modeling pilot’s sequential maneuvering decisions by a multistage influence diagram[J]. Journal of Guidance, Control, and Dynamics200427(4): 665-677.
10 EKLUND J M, SPRINKLE J, SASTRY S S. Switched and symmetric pursuit/evasion games using online model predictive control with application to autonomous aircraft[J]. IEEE Transactions on Control Systems Technology201220(3): 604-620.
11 HUANG C Q, DONG K S, HUANG H Q, et al. Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization[J]. Journal of Systems Engineering and Electronics201829(1): 86-97.
12 MCGREW J S, HOW J P, WILLIAMS B, et al. Air-combat strategy using approximate dynamic programming[J]. Journal of Guidance, Control, and Dynamics201033(5): 1641-1654.
13 ZHANG H P, HUANG C Q. Maneuver decision-making of deep learning for UCAV thorough azimuth angles[J]. IEEE Access20208: 12976-12987.
14 YANG Q M, ZHANG J D, SHI G Q, et al. Maneuver decision of UAV in short-range air combat based on deep reinforcement learning?[J]. IEEE Access20208: 363-378.
15 ERNEST N, CARROLL D, SCHUMACHER C, et al.Genetic fuzzy based artificial intelligence for unmanned combat aerial vehicle control in simulated air combat missions?[J]. Journal of Defense Management20166(1): 144.
16 DUAN H B, LI P, YU Y X. A predator-prey particle swarm optimization approach to multiple UCAV air combat modeled by dynamic game theory?[J]. IEEE/CAA Journal of Automatica Sinica20152(1): 11-18.
17 HAN T, WANG X F, LIANG Y J, et al. Study on UCAV robust maneuvering decision in automatic air combat based on target accessible domain[J]. Journal of Physics: Conference Series20191213(5): 052004.
18 HO E, RAJAGOPALAN A, SKVORTSOV A, et al. Game theory in defence applications: A review[J]. Sensors202222(3): 1032.
19 钟友武, 杨凌宇, 柳嘉润, 等. 基于智能微分对策的自主机动决策方法研究[J]. 飞行力学200826(6): 29-33.
  ZHONG Y W, YANG L Y, LIU J R, et al. Method of autonomous maneuver decision based on intelligent differential game[J]. Flight Dynamics200826(6): 29-33 (in Chinese).
20 王义宁, 姜玉宪. 空战决策中的智能微分对策法[J]. 飞行力学200321(1): 66-70.
  WANG Y N, JIANG Y X. An intelligent differential gameon air combat decision[J]. Flight Dynamics200321(1): 66-70 (in Chinese).
21 LEE B Y, HAN S, PARK H J, et al. One-versus-one air-to-air combat maneuver generation based on the differential game[C]∥ Proceedings of the 2016 Congress of the International Council of the Aeronautical Sciences. ICAS, 2016: 1-7.
22 HORIE K, CONWAY B A. Optimal fighter pursuit-evasion maneuvers found via two-sided optimization[J]. Journal of Guidance, Control, and Dynamics200629(1): 105-112.
23 XU G Y, WEI S N, ZHANG H M. Application of situation function in air combat differential games[C]∥ 2017 36th Chinese Control Conference (CCC). Piscataway: IEEE Press, 2017: 5865-5870.
24 GARCIA E, MOLLA V, CASBEER D W, et al. Strategies for defending a coastline against multiple attackers[C]∥ 2019 IEEE 58th Conference on Decision and Control (CDC). Piscataway: IEEE Press, 2019: 7319-7324.
25 VIRTANEN K, KARELAHTI J, RAIVIO T. Modeling air combat by a moving horizon influence diagram game[J]. Journal of Guidance, Control, and Dynamics200629(5): 1080-1091.
26 ZHONG L, TONG M A, ZHONG W, et al. Sequential maneuvering decisions based on multi-stage influence diagram in air combat[J]. Journal of Systems Engineering and Electronics200718(3): 551-555.
27 LU H C, WU B Y, CHEN J Q. Fighter equipment contribution evaluation based on maneuver decision?[J]. IEEE Access20219: 132241-132254.
28 PAN Q, ZHOU D Y, HUANG J C, et al. Maneuver decision for cooperative close-range air combat based on state predicted influence diagram[C]∥ 2017 IEEE International Conference on Information and Automation (ICIA). Piscataway: IEEE Press, 2017: 726-731.
29 ZHOU S Y, WANG Z J, FAN G, et al. Collaborative maneuvering decision based on multi-layer influence diagram group decision-making[M]∥ Advances in Transdisciplinary Engineering. Netherlands: IOS Press, 2024:345-357.
30 钟麟, 佟明安, 钟卫. 影响图对策在多机协同空战中的应用[J]. 北京航空航天大学学报200733(4): 450-453.
  ZHONG L, TONG M A, ZHONG W. Application of multistage influence diagram game theory for multiple cooperative air combat[J]. Journal of Beijing University of Aeronautics and Astronautics200733(4): 450-453 (in Chinese).
31 GENG W X, KONG F E, MA D Q. Study on tactical decision of UAV medium-range air combat[C]∥ The 26th Chinese Control and Decision Conference (2014 CCDC). Piscataway: IEEE Press, 2014: 135-139.
32 WANG X F, WANG B S. Situation assessment method based on Bayesian network and intuitionistic fuzzy reasoning[J]. System Engineering and Electronics200931(11): 2742-2746.
33 刘守业. 非完备信息下无人机空战决策与导引方法研究[D]. 沈阳: 沈阳航空航天大学, 2022: 3-28.
  LIU S Y. Decision-making and guidance method of unmanned aerial vehicle under incomplete information[D]. Shenyang: Shenyang Aerospace University, 2022: 3-28 (in Chinese).
34 PIAO H Y, SUN Z X, MENG G L, et al. Beyond-visual-range air combat tactics auto-generation by reinforcement learning[C]∥ 2020 International Joint Conference on Neural Networks (IJCNN). Piscataway: IEEE Press, 2020: 1-8.
35 张强, 杨任农, 俞利新, 等. 基于Q-network强化学习的超视距空战机动决策[J]. 空军工程大学学报201819(6): 8-14.
  ZHANG Q, YANG R N, YU L X, et al. BVR air combat maneuvering decision by using Q-network reinforcement learning[J]. Journal of Air Force Engineering University201819(6): 8-14 (in Chinese).
36 YANG Q M, ZHU Y, ZHANG J D, et al. UAV air combat autonomous maneuver decision based on DDPG algorithm[C]?∥ 2019 IEEE 15th International Conference on Control and Automation (ICCA). Piscataway: IEEE Press, 2019: 37-42.
37 KONG W R, ZHOU D Y, YANG Z, et al. UAV autonomous aerial combat maneuver strategy generation with observation error based on state-adversarial deep deterministic policy gradient and inverse reinforcement learning[J]. Electronics20209(7): 1121.
38 黄长强, 赵克新, 韩邦杰, 等. 一种近似动态规划的无人机机动决策方法[J]. 电子与信息学报201840(10): 2447-2452.
  HUANG C Q, ZHAO K X, HAN B J, et al. Maneuvering decision-making method of UAV based on approximate dynamic programming[J]. Journal of Electronics & Information Technology201840(10): 2447-2452 (in Chinese).
39 MA Y F, MA X L, SONG X. A case study on air combat decision using approximated dynamic programming[J]. Mathematical Problems in Engineering2014(4): 183401.
40 梅丹, 刘锦涛, 高丽. 基于近似动态规划与零和博弈的空战机动决策[J]. 兵工自动化201736(3): 35-39.
  MEI D, LIU J T, GAO L. Maneuver decision of air combat based on approximate dynamic programming and zero-sum game?[J]. Ordnance Industry Automation201736(3): 35-39 (in Chinese).
41 HU Z C, GAO P, WANG F. Research on autonomous maneuvering decision of UCAV based on approximate dynamic programming[C]∥ 2019 International Conference on Image and Video Processing, and Artificial Intelligence. NewYork: SPIE, 2019: 636-641.
42 姜龙亭, 寇雅楠, 王栋, 等. 改进近似动态规划法的攻击占位决策[J]. 火力与指挥控制201944(7): 135-141.
  JIANG L T, KOU Y N, WANG D, et al. Attack placeholder decision based on improved approximate dynamic programming[J]. Fire Control & Command Control201944(7): 135-141 (in Chinese).
43 CRUMPACKER J B, ROBBINS M J, JENKINS P R. An approximate dynamic programming approach for solving an air combat maneuvering problem[J]. Expert Systems with Applications2022203: 117448.
44 ARULKUMARAN K, DEISENROTH M P, BRUNDAGE M, et al. Deep reinforcement learning: A brief survey?[J]. IEEE Signal Processing Magazine201734(6): 26-38.
45 SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature2016529: 484-489.
46 SILVER D, SCHRITTWIESER J, SIMONYAN K, et al. Mastering the game of Go without human knowledge[J]. Nature2017550: 354-359.
47 VINYALS O, BABUSCHKIN I, CZARNECKI W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. Nature2019575: 350-354.
48 BERNER C, BROCKMAN G, CHAN B, et al. Dota 2 with large scale deep reinforcement learning[DB/OL]. arXiv preprint1912.06680, 2019.
49 NGUYEN T T, NGUYEN N D, NAHAVANDI S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications?[J]. IEEE Transactions on Cybernetics202050(9): 3826-3839.
50 LYU L, SHEN Y, ZHANG S C. The advance of reinforcement learning and deep reinforcement learning[C]∥ 2022 IEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA). Piscataway: IEEE Press, 2022: 644-648.
51 SARKAR N I, GUL S. Artificial intelligence-based autonomous UAV networks: A survey[J]. Drones20237(5): 322.
52 BAYERLEIN H, THEILE M, CACCAMO M, et al. Multi-UAV path planning for wireless data harvesting with deep reinforcement learning[J]. IEEE Open Journal of the Communications Society20212: 1171-1187.
53 ZHU B T, BEDEER E, NGUYEN H H, et al. UAV trajectory planning in wireless sensor networks for energy consumption minimization by deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology202170(9): 9540-9554.
54 WANG X, CHEN Y D, ZHU W W. A survey on curriculum learning?[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence202244(9): 4555-4576.
55 WANG Y B, JIANG T S, LI Y J. A hierarchical reinforcement learning method on multi UCAV air combat[C]?∥ 2021 International Conference on Neural Networks, Information and Communication Engineering. New York: SPIE, 202111933: 117-123.
56 WANG L, WANG K Z, PAN C H, et al. Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing[J]. IEEE Transactions on Cognitive Communications and Networking20217(1): 73-84.
57 ZHAO W L, MENG Z J, WANG K P, et al. Hierarchical active tracking control for UAVs via deep reinforcement learning?[J]. Applied Sciences202111(22): 10595.
58 LI B H, WU Y J. Path planning for UAV ground target tracking via deep reinforcement learning?[J]. IEEE Access20208: 29064-29074.
59 LI B, GAN Z G, CHEN D Q, et al. UAV maneuvering target tracking in uncertain environments based on deep reinforcement learning and meta-learning?[J]. Remote Sensing202012(22): 3789.
60 LI Y, HAN W, WANG Y Q. Deep reinforcement learning with application to air confrontation intelligent decision-making of manned/unmanned aerial vehicle cooperative system?[J]. IEEE Access20208: 67887-67898.
61 WU L Z, WANG C, ZHANG P P, et al. Deep reinforcement learning with corrective feedback for autonomous UAV landing on a mobile platform[J]. Drones20226(9): 238.
62 XIE J Y, PENG X D, WANG H J, et al. UAV autonomous tracking and landing based on deep reinforcement learning strategy[J]. Sensors202020(19): 5630.
63 XU D, GUO Y X, YU Z Y, et al. PPO-Exp: Keeping fixed-wing UAV formation with deep reinforcement learning[J]. Drones20227(1): 28.
64 ZHAO W W, CHU H R, MIAO X K, et al. Research on the multiagent joint proximal policy optimization algorithm controlling cooperative fixed-wing UAV obstacle avoidance[J]. Sensors202020(16): 4546.
65 ZHAO Y, CHEN Y F, ZHEN Z Y, et al. Multi-weapon multi-target assignment based on hybrid genetic algorithm in uncertain environment[J]. International Journal of Advanced Robotic Systems202017(2): 1729881420905922.
66 ZHAO X R, YANG R N, ZHANG Y, et al. Deep reinforcement learning for intelligent dual-UAV reconnaissance mission planning[J]. Electronics202211(13): 2031.
67 YUE L F, YANG R N, ZHANG Y, et al. Deep reinforcement learning for UAV intelligent mission planning[J]. Complexity2022(1): 3551508.
68 GUO T, JIANG N, LI B Y, et al. UAV navigation in high dynamic environments: A deep reinforcement learning approach[J]. Chinese Journal of Aeronautics202134(2): 479-489.
69 HU J W, WANG L H, HU T M, et al. Autonomous maneuver decision making of dual-UAV cooperative air combat based on deep reinforcement learning[J]. Electronics202211(3): 467.
70 WANG X W, WANG Y H, SU X C, et al. Deep reinforcement learning-based air combat maneuver decision-making: Literature review, implementation tutorial and future direction[J]. Artificial Intelligence Review202357(1): 1.
71 CAO Y, KOU Y X, LI Z W, et al. Autonomous maneuver decision of UCAV air combat based on double deep Q network algorithm and stochastic game theory[J]. International Journal of Aerospace Engineering2023(1): 3657814.
72 ZHANG X B, LIU G Q, YANG C J, et al. Research on air confrontation maneuver decision-making method based on reinforcement learning[J]. Electronics20187(11): 279.
73 PIAO H Y, HAN Y, CHEN H C, et al. Complex relationship graph abstraction for autonomous air combat collaboration: A learning and expert knowledge hybrid approach?[J]. Expert Systems with Applications2023215: 119285.
74 SUN Z X, PIAO H Y, YANG Z, et al. Multi-agent hierarchical policy gradient for air combat tactics emergence via self-play[J]. Engineering Applications of Artificial Intelligence202198: 104112.
75 LI Y F, SHI J P, JIANG W, et al. Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MS-DDQN algorithm[J]. Defence Technology202218(9): 1697-1714.
76 ZHANG H P, ZHOU H, WEI Y J, et al. Autonomous maneuver decision-making method based on reinforcement learning and Monte Carlo tree search[J]. Frontiers in Neurorobotics202216: 996412.
77 YANG K B, DONG W H, CAI M, et al. UCAV air combat maneuver decisions based on a proximal policy optimization algorithm with situation reward shaping[J]. Electronics202211(16): 2602.
78 ZHENG Z Q, DUAN H B. UAV maneuver decision-making via deep reinforcement learning for short-range air combat[J]. Intelligence & Robotics20233(1): 76-94.
79 杨书恒, 张栋, 熊威, 等. 基于可解释性强化学习的空战机动决策方法[J]. 航空学报202445(18): 329922.
  YANG S H, ZHANG D, XIONG W, et al. Decision-making method for air combat maneuver based on explainable reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica202445(18): 329922 (in Chinese).
80 谢建峰, 杨啟明, 戴树岭, 等. 基于强化遗传算法的无人机空战机动决策研究[J]. 西北工业大学学报202038(6): 1330-1338.
  XIE J F, YANG Q M, DAI S L, et al. Air combat maneuver decision based on reinforcement genetic algorithm[J]. Journal of Northwestern Polytechnical University202038(6): 1330-1338 (in Chinese).
81 周德云, 李锋, 蒲小勃, 等. 基于遗传算法的飞机战术飞行动作决策[J]. 西北工业大学学报200220(1): 109-112.
  ZHOU D Y, LI F, PU X B, et al. On improving tactical planning in air combat in P.R.China with genetic algorithm[J]. Journal of Northwestern Polytechnical University200220(1): 109-112 (in Chinese).
82 王杰, 丁达理, 董康生, 等. UCAV自主空战战术机动动作建模与轨迹生成[J]. 火力与指挥控制201843(12): 42-49.
  WANG J, DING D L, DONG K S, et al. UCAV autonomous air combat tactical maneuvering modeling and trajectory generation[J]. Fire Control & Command Control201843(12): 42-49 (in Chinese).
83 张涛, 于雷, 周中良, 等. 基于变权重伪并行遗传算法的空战机动决策[J]. 飞行力学201230(5): 470-474.
  ZHANG T, YU L, ZHOU Z L, et al. Decision-making for aircombat maneuvering based on variable weight pseudo-parallel genetical algorithm?[J]. Flight Dynamics201230(5): 470-474 (in Chinese).
84 BURGIN G H, SIDOR L B. Rule-based air combat simulation: NASA-CR-4160[R]. Washington,D.?C.: National Aeronautics and Space Administration, 1988.
85 王锐平, 高正红. 无人机空战仿真中基于机动动作库的决策模型[J]. 飞行力学200927(6): 72-75, 79.
  WANG R P, GAO Z H. Research on decision system in air combat simulation using maneuver library[J]. Flight Dynamics200927(6): 72-75, 79 (in Chinese).
86 王刚, 雷英杰, 何晶. 空战决策指挥引导专家系统[J]. 空军工程大学学报(自然科学版)20023(1): 11-13.
  WANG G, LEI Y J, HE J. Interception guidance expert system for airfight decision[J]. Journal of Air Force Engineering University (Natural Science Edition)20023(1): 11-13 (in Chinese).
87 谭目来, 丁达理, 谢磊, 等. 基于模糊专家系统与IDE算法的UCAV逃逸机动决策[J]. 系统工程与电子技术202244(6): 1984-1993.
  TAN M L, DING D L, XIE L, et al. UCAV escape maneuvering decision based on fuzzy expert system and IDE algorithm?[J]. Systems Engineering and Electronics202244(6): 1984-1993 (in Chinese).
88 QIAN C X, ZHANG X B, LI L, et al. H3E: Learning air combat with a three-level hierarchical framework embedding expert knowledge[J]. Expert Systems with Applications2024245: 123084.
89 CHAI R Q, TSOURDOS A, SAVVARIS A, et al. Solving constrained trajectory planning problems using biased particle swarm optimization[J]. IEEE Transactions on Aerospace and Electronic Systems202157(3): 1685-1701.
90 RUAN W Y, DUAN H B, DENG Y M. Autonomous maneuver decisions via transfer learning pigeon-inspired optimization for UCAVs in dogfight engagements[J]. IEEE/CAA Journal of Automatica Sinica20229(9): 1639-1657.
91 TAN M L, TANG A D, DING D L, et al. Autonomous air combat maneuvering decision method of UCAV based on LSHADE-TSO-MPC under enemy trajectory prediction[J]. Electronics202211(20): 3383.
Outlines

/