Abstract:

In the case of two Unmanned Combat Aerial Vehicles （UCAVs） cooperative air combat with local observation， there are the problems such as hard-to-design collaborative rewards， low collaboration efficiency and poor decision-making effect. To solve these problems， an intelligent maneuver decision-making method is proposed based on the improved the Multi-Agent Deep Deterministic Policy Gradient （MADDPG） with hybrid hyper network. A Centralized Training with Decentralized Execution （CTDE） architecture is adopted to meet the training requirements of global coordinated maneuvering decision in the situation of single agent with local observation. A reward function is designed for the UCAV agent by considering both the local reward of fast guidance for obtaining attack advantage and the global reward for winning air combat. Then， a hybrid hyper network is introduced to mix the estimated Q values of each agent monotonically and nonlinearly to obtain the global policy value function. By using the global policy value function， the decentralized Actor network update parameters to solve the problem of credit assignment in multi-agent deep reinforcement learning. Simulation results show that compared with the traditional MADDPG method， the proposed method can produce the optimal global cooperative maneuver commands for achieving better coordination performance， and can obtain a higher winning rate with the same agent opponent.

Key words: unmanned combat aerial vehicle, air combat maneuvering decision, Multi-Agent Deep Deterministic Policy Gradient (MADDPG), hybrid hyper network, centralized training with decentralized execution