航空学报 > 2021, Vol. 42 Issue (11): 524688-524688   doi: 10.7527/S1000-6893.2020.24688

深度确定性策略梯度算法用于无人飞行器控制

黄旭1,2, 柳嘉润1,2, 贾晨辉1,2, 王昭磊1,2, 张隽1,2   

  1. 1. 北京航天自动控制研究所, 北京 100854;
    2. 宇航智能控制技术国家级重点实验室, 北京 100854
  • 收稿日期:2020-08-31 修回日期:2020-09-04 发布日期:2020-09-17
  • 通讯作者: 柳嘉润 E-mail:jiarunliu@163.com
  • 基金资助:
    国家自然科学基金(61773341)

Deep deterministic policy gradient algorithm for UAV control

HUANG Xu1,2, LIU Jiarun1,2, JIA Chenhui1,2, WANG Zhaolei1,2, ZHANG Jun1,2   

  1. 1. Beijing Aerospace Automatic Control Institute, Beijing 100854, China;
    2. National Key Laboratory of Science and Technology on Aerospace Intelligent Control, Beijing 100854, China
  • Received:2020-08-31 Revised:2020-09-04 Published:2020-09-17
  • Supported by:
    National Natural Science Foundation of China (61773341)

摘要: 对深度确定性策略梯度算法训练智能体学习小型无人飞行器的飞行控制策略进行了探索研究。以多数据帧的速度、位置和姿态角等信息作为智能体的观察状态,舵摆角和发动机推力指令作为智能体的输出动作,飞行器的非线性模型和飞行环境作为智能体的学习环境。智能体在与环境交互过程中除了获得包含误差信息的密集惩罚外,也有达成一定目标的稀疏奖励,该设计有效提高了飞行数据的样本多样性,增强了智能体的学习效率。最后智能体实现了从位置、速度和姿态角等信息到控制量的端到端飞行控制,并进行了变航迹点、模型参数拉偏、注入扰动和故障条件下的飞行控制仿真,结果表明智能体除了能有效完成训练任务外,还能应对多种训练时未学习的飞行任务,具有优秀的泛化能力和鲁棒性,该方法具有一定的研究价值和工程参考价值。

关键词: 深度确定性策略梯度, 小型无人飞行器, 飞行控制, 端到端, 稀疏奖励

Abstract: The deep deterministic policy gradient algorithm is used to train the agent to learn the flight control strategy of a small UAV. The velocity, position and attitude angle of multi data frames are taken as the observation state of the agent, the rudder deflection angle and engine thrust command the output actions of the agent, and the nonlinear model and flight environment of the UAV the learning environment of the agent. In the interaction process between the agent and the environment, sparse rewards are provided to achieve certain goals, in addition to the dense punishment including error information, thereby effectively improving the diversity of flight data samples and enhancing the learning efficiency of the agent. The agent finally realizes the end-to-end flight control from the position, velocity and attitude angle to the control variables. In addition, the flight control simulations under the conditions of variable track point, model parameter deviation, disturbance and fault are carried out. Simulation results show that the agent can not only effectively complete the training task, but also deal with a variety of flight tasks not learned during training, showing excellent generalization ability and exhibiting certain research value and engineering reference value of the method.

Key words: deep deterministic policy gradient, small UAV, flight control, end to end, sparse reward

中图分类号: