基于多智能体强化学习的空间机械臂轨迹规划

doi:10.7527/S1000-6893.2020.24151

论文

本期目录 | 过刊浏览 | 高级检索

前一篇 | 后一篇

基于多智能体强化学习的空间机械臂轨迹规划

赵毓, 管公顺, 郭继峰, 于晓强, 颜鹏

哈尔滨工业大学航天学院, 哈尔滨 150001

收稿日期:2020-04-28 修回日期:2020-05-21 发布日期:2020-07-10
通讯作者: 郭继峰 E-mail:guojifeng@hit.edu.cn
基金资助:
国家自然科学基金（61973101）；航空科学基金（20180577005）

Trajectory planning of space manipulator based on multi-agent reinforcement learning

ZHAO Yu, GUAN Gongshun, GUO Jifeng, YU Xiaoqiang, YAN Peng

School of Astronautics, Harbin Institute of Technology, Harbin 150001, China

Received:2020-04-28 Revised:2020-05-21 Published:2020-07-10
Supported by:
National Natural Science Foundation of China (61973101); Aeronautical Science Foundation of China (20180577005)

摘要/Abstract

摘要： 针对某型六自由度（DOF）空间漂浮机械臂对运动目标捕捉场景，开展了基于深度强化学习的在线轨迹规划方法研究。首先给出了机械臂DH （Denavit-Hartenberg）模型，考虑组合体力学耦合特性建立了多刚体运动学和动力学模型。然后提出了一种改进深度确定性策略梯度算法，以各关节为决策智能体建立了多智能体自学习系统。而后建立了"线下集中学习，线上分布执行"的空间机械臂对匀速直线运动目标捕捉训练系统，构建以目标相对距离和总操作时间为参数的奖励函数。最后通过数学仿真验证，实现了机械臂对各向匀速运动目标的快速捕捉，平均完成耗时5.4 s。与传统基于随机采样的规划算法对比，本文提出的自主决策运动规划方法求解速度和鲁棒性更优。

关键词: 机械臂, 轨迹规划, 多智能体, 策略梯度, 在轨捕捉

Abstract: An online self-learning trajectory planning method based on the deep reinforcement learning is studied for a six Degree-of-Freedom (DOF) space floating manipulator to capture moving objects. The DH(Denavit-Hartenberg) model of the manipulator is presented, and the kinematic and dynamic models of multi-rigid bodies established considering the mechanical coupling characteristics of the combination. An improved deep determination policy gradient algorithm is further proposed, and a multi-agent self-learning system established with each joint as a decision-making agent. Additionally, a training model of the space manipulator is built based on "offline centralized learning and online distributed execution", constructing a reward function with the variables of the target relative distance and the total operation time. Simulation results show that the robot can capture the moving target rapidly with average time of 5.4 s. Compared with the traditional planning algorithm based on random sampling, the autonomous decision-making motion planning method proposed in this paper exhibits better solution speed and robustness.

Key words: manipulators, trajectory planning, multi-agent, policy gradient, on orbit capture

中图分类号:

V447

赵毓, 管公顺, 郭继峰, 于晓强, 颜鹏. 基于多智能体强化学习的空间机械臂轨迹规划[J]. 航空学报, 2021, 42(1): 524151-524151.

ZHAO Yu, GUAN Gongshun, GUO Jifeng, YU Xiaoqiang, YAN Peng. Trajectory planning of space manipulator based on multi-agent reinforcement learning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(1): 524151-524151.

参考文献

[1] 王平, 郭继峰, 史晓宁, 等. 服务航天器近距离操作四维运动规划方法研究[J]. 控制与决策, 2010, 25(10):1519-1522. WANG P, GUO J F, SHI X N, et al. Research on four-dimensional motion planning of on-orbit servicing autonomous spacecraft[J]. Control and Decision, 2010, 25(10):1519-1522(in Chinese).
[2] 介党阳, 陆浩然, 吴晗玲, 等. 空间大型机械臂系统载运轨迹优化方法[J]. 航空学报, 2018, 39(S1):111-119. JIE D Y, LU H R, WU H L, et al. Transporting trajectory optimization method for large space manipulator system[J]. Acta Aeronautica et Astronautica Sinica, 2018, 39(S1):111-119(in Chinese).
[3] 张彤彤. 空间机械臂抓取非合作目标的轨迹规划与智能控制研究[D]. 哈尔滨:哈尔滨工业大学, 2019:15-20. ZHANG T T. Research on trajectory planning and intelligent control of space manipulator capture for noncooperative targets[D]. Harbin:Harbin Institute of Technology, 2019:15-20(in Chinese).
[4] JOSEPH N P. A path forward to better space security:Finding new solutions to space debris, space situational awareness and space traffic management[J]. Journal of Space Safety Engineering, 2019, 6(2):92-100.
[5] XU R N, LUO J J, WANG M M. Kinematic and dynamic manipulability analysis for free-floating space robots with closed chain constraints[J]. Robotics and Autonomous Systems, 2020, 130(4):1-9.
[6] YOSHIDA K, HASHIZUME K, ABIKO S. Zero reaction maneuver:Flight validation with ETS-VⅡ space robot and extension to kinematically redundant arm[C]//IEEE International Conference on Robotics & Automation. Piscataway:IEEE Press, 2001:441-446.
[7] 徐文福. 空间机器人目标捕获的路径规划与实验研究[D]. 哈尔滨:哈尔滨工业大学, 2007:126-133. XU W F. Path planning and experiment study of space robot for target capturing[D]. Harbin:Harbin Institute of Technology, 2007:126-133(in Chinese).
[8] 崔浩, 戈新生. 自由漂浮空间机器人运动规划的多项式插值法[J]. 北京信息科技大学学报, 2019, 34(4):17-23,29. CUI H, GE X S. Polynomial interpolation method for path optimization of free-floating space robots[J]. Journal of Beijing Information Science & Technology University, 2019, 34(4):17-23,29(in Chinese).
[9] 刘宏, 刘宇, 姜力. 空间机器人及其遥操作[M]. 哈尔滨:哈尔滨工业大学出版社, 2012:174-192. LIU H, LIU Y, JIANG L. Space robot and its teleoperation[M]. Harbin:Harbin Institute of Technology Press, 2012:174-192(in Chinese).
[10] 王明, 黄攀峰, 常海涛, 等. 基于机械臂耦合力矩评估的组合航天器姿态协调控制[J]. 机器人, 2015, 37(1):25-34. WANG M, HUANG P F, CHANG H T, et al. Coordinated attitude control of combined spacecraft based on estimated coupling torque of manipulator[J]. Robot, 2015, 37(1):25-34(in Chinese).
[11] CONKUR E S. Path planning using potential fields for highly redundant manipulators[J]. Robotics and Autonomous Systems, 2005, 52(2-3):209-228.
[12] LAVALLE S M. Rapidly-exploring random trees:A new tool for path planning:TR98-11[R]. Ames:Iowa State University, 1998.
[13] LARS L, JONGHWA K, MICHAEL K, et al. Automatic path planning of industrial robots comparing sampling-based and computational intelligence methods[J]. Procedia Manufacturing, 2017, 11(6):241-248.
[14] LYDIA E K, PETR S, JEAN L, et al. Probabilistic roadmaps for path planning in high-dimensional configuration spaces[J]. IEEE Transactions on Robotics and Automation, 1996, 4(12):566-580.
[15] CAO X M, ZOU X J, JIA C Y, et al. RRT-based path planning for an intelligent litchi-picking manipulator[J]. Computers and Electronics in Agriculture, 2019, 156(1):105-118.
[16] 关英姿, 宋春林, 董惠娟. 空间自由漂浮机器人对运动目标抓捕的路径规划[J]. 机器人, 2017, 39(6):803-811. GUAN Y Z, SONG C L, DONG H J. Path planning of the free-floating manipulator for capturing a moving target[J]. Robot, 2017, 39(6):803-811(in Chinese).
[17] GUPTA J K, EGOROV M, KOCHENDERFER M. Cooperative multi-agent control using deep reinforcement learning[C]//International Conference on Autonomous Agents and Multi-Agent Systems. Heidelberg:Springer, 2017:66-83.
[18] MICHAEL S, RONAY A, THOMAS H. A survey of the advancing use and development of machine learning in smart manufacturing[J]. Journal of Manufacturing Systems, 2018, 48(7):170-179.
[19] 赵辉. 基于Q学习算法的机械臂轨迹规划研究[D]. 大庆:东北石油大学, 2013:23-29. ZHAO H. Research on the manipulator trajectory planning based on Q-learning[D]. Daqing:Northeast Petroleum University, 2013:23-29(in Chinese).
[20] 徐帷, 卢山. 基于Sarsa(λ)强化学习的空间机械臂路径规划研究[J]. 宇航学报, 2019, 40(4):435-443. XU W, LU S. Analysis of space manipulator route planning based on Sarsa (λ) reinforcement learning[J]. Journal of Astronautics, 2019, 40(4):435-443(in Chinese).
[21] 刘钱源. 基于深度强化学习的双臂机器人物体抓取[D]. 威海:山东大学, 2019:24-36. LIU Q Y. Deep reinforcement learning based object grasping of dual-arm robot[D]. Weihai:Shandong University, 2019:24-36(in Chinese).
[22] 陈建华. 基于深度强化学习的机械臂运动规划研究[D]. 秦皇岛:燕山大学, 2019:42-57. CHEN J H. Research on motion planning of robot arm based on deep reinforcement learning[D]. Qinhuangdao:Yanshan University, 2019:42-57(in Chinese).
[23] LOWR R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Annual Conference on Neural Information Processing Systems, 2017:6379-6390.
[24] 滕军. 智能工业机器人的环境感知与运动规划[D]. 哈尔滨:哈尔滨工业大学, 2019:68-78. TENG J. Environment perception and motion planning for intelligent industrial robot[D]. Harbin:Harbin Institute of Technology, 2019:68-78(in Chinese).
[25] 曾祥鑫. 自由漂浮空间机器人路径规划及控制方法研究[D]. 哈尔滨:哈尔滨工业大学, 2018:33-55. ZENG X X. Research on path planning and control method for free-floating space robot[D]. Harbin:Harbin Institute of Technology, 2018:33-55(in Chinese).
[26] 刘延栋. 基于DDPG强化学习的移动机器人路径规划[D]. 呼和浩特:内蒙古工业大学, 2019:11-31. LIU Y D. Mobile robot path planning based on DDPG reinforcement learning network[D]. Hohhot:Inner Mongolia University, 2019:11-31(in Chinese).

E-mail：hkxb@buaa.edu.cn

关于我们

期刊社服务

专业学科

封面文章

友情链接

主管单位：中国科学技术协会主办单位：中国航空学会北京航空航天大学

基于多智能体强化学习的空间机械臂轨迹规划

Trajectory planning of space manipulator based on multi-agent reinforcement learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张薇, 何若俊. 面向物联网数据收集的无人机自主路径规划[J]. 航空学报, 2024, 45(8): 329054-329054-1.
[2]	董豪泽, 陈昱达, 刘丹, 李忠奎. 障碍物空间下分布式轨迹规划的死锁破解[J]. 航空学报, 2023, 44(S2): 729771-729771.
[3]	王雪鉴, 文永明, 石晓荣, 张宁宁, 刘洁玺. 多智能体多耦合任务混合式智能决策架构设计[J]. 航空学报, 2023, 44(S2): 729770-729770.
[4]	邓有朋, 范佳宣, 郑岩, 王振亚, 吕勇梁, 李雨霄. 不完全信息下多智能体对手建模[J]. 航空学报, 2023, 44(S2): 729782-729782.
[5]	刘哲, 张羲格, 韦常柱, 崔乃刚. 亚轨道飞行器再入轨迹高精度自适应凸规划[J]. 航空学报, 2023, 44(S2): 729430-729430.
[6]	范之琳, 杨洪勇, 韩艺琳. 基于强化学习的多智能体系统目标围捕控制[J]. 航空学报, 2023, 44(S1): 727487-727487.
[7]	于连波, 曹品钊, 石亮, 连捷, 王东. 基于改进冲突搜索的多智能体路径规划算法[J]. 航空学报, 2023, 44(S1): 727648-727648.
[8]	马亚杰, 王娟, 姜斌, 龚建业. 一种无人机⁃无人车编队系统容错控制方法[J]. 航空学报, 2023, 44(8): 327216-327216.
[9]	符小卫, 徐哲, 朱金冬, 王楠. 基于PER-MATD3的多无人机攻防对抗机动决策[J]. 航空学报, 2023, 44(7): 327083-327083.
[10]	高树一, 林德福, 郑多, 胡馨予. 针对集群攻击的飞行器智能协同拦截策略[J]. 航空学报, 2023, 44(18): 328301-328301.
[11]	董磊, 陈泓兵, 陈曦, 赵长啸. 基于DQN的单一飞行员驾驶模式分布式多智能体联盟任务分配策略[J]. 航空学报, 2023, 44(13): 327895-327895.
[12]	牛皓飞, 蔡庆中, 李健, 杨功流. 基于图优化的通信受限环境下协同导航方法[J]. 航空学报, 2023, 44(11): 327342-327342.
[13]	冉庆波, 肖鸿, 杨富鸿, 段玉岗. 含孔曲面自动铺丝轨迹规划算法[J]. 航空学报, 2022, 43(9): 425602-425602.
[14]	孙杨, 昌敏, 白俊强. 微小型四旋翼无人机垂面栖停轨迹规划与控制[J]. 航空学报, 2022, 43(9): 325756-325756.
[15]	符小卫, 王辉, 徐哲. 基于DE-MADDPG的多无人机协同追捕策略[J]. 航空学报, 2022, 43(5): 325311-325311.