Abstract: Intelligent air combat is a hot research topic among countries with strong military in the world. In order to solve the maneuver decision problem of air combat Beyond Visual Range (BVR), we propose the hierarchical decision algorithm based on Deep Reinforcement Learning. We use the maneuver set appropriate to the BVR air combat and we control the trajectory and the attitude of the aircraft with flight control law in the decision algorithm. In order to expand the action space of the model and increase its ability of decision-making, we hierarchize the action space and model it as the multi-discrete one. Aimed at the problem of sparse reward in air combat, we design a set of reward function taking the position advantage, weapon launching, weapon threat and other factors into consideration, which can guide the agent to converge to the optimal policy. We also build a complete digital-twin simulation environment for air combat and an expert system. The decision algorithm is trained in the environment and is evaluated by fighting with the expert system. The experiment results indicates that the decision algorithm we propose has the ability to make autonomous and flexible decision in BVR air combat based on current situation and can attack periodically. It has some advantage against the expert system.

Key words: air combat beyond visual range, intelligent decision, deep reinforcement learning, PPO, maneuver, hierar-chical decision