基于启发强化学习的大规模ADR任务优化方法

doi:10.7527/S1000-6893.2020.24354

电子电气工程与控制

本期目录 | 过刊浏览 | 高级检索

前一篇 | 后一篇

基于启发强化学习的大规模ADR任务优化方法

杨家男¹, 侯晓磊¹, HU Yu Hen², 刘勇¹, 潘泉¹, 冯乾¹

1. 西北工业大学自动化学院, 西安 710129;
2. 美国威斯康星大学麦迪逊分校电气与计算机工程系, 麦迪逊 53706

收稿日期:2020-06-02 修回日期:2020-09-12 发布日期:2020-10-10
通讯作者: 杨家男 E-mail:yang_jia_nan@mail.nwpu.edu.cn
基金资助:
国家自然科学基金（61703343，61790552）；陕西省自然科学基金（2018JQ6070）；中央高校基本科研业务费（3102018JCC003）

Heuristic enhanced reinforcement learning method for large-scale multi-debris active removal mission planning

YANG Jianan¹, HOU Xiaolei¹, HU Yu Hen², LIU Yong¹, PAN Quan¹, FENG Qian¹

1. College of Automation, Northwestern Polytechnical University, Xi'an 710129, China;
2. Department of Electrical and Computer Engineering, University of Wisconsin-Madison, Madison 53706, USA

Received:2020-06-02 Revised:2020-09-12 Published:2020-10-10
Supported by:
National Natural Science Foundation of China (61703343, 61790552); Natural Science Foundation of Shaanxi (2018JQ6070); The Fundamental Research Funds for Central Universities (3102018JCC003)

摘要/Abstract

摘要： 随着航天事业的蓬勃发展，空间碎片尤其是低轨碎片已成为航天任务不可忽视的威胁。考虑到碎片清除的紧迫性和成本，低轨多碎片主动清除（ADR）技术成为缓解现状的必要手段。针对大规模多碎片主动清除任务规划问题，首先，基于任务规划的最大收益模型，提出一种强化学习（RL）优化方法，并依照强化学习框架定义了该问题的状态、动作以及收益函数；其次，基于高效启发因子，提出一种专用的改进蒙特卡罗树搜索（MCTS）算法，该算法使用MCTS算法作为内核，加入高效启发算子以及强化学习迭代过程；最后，在铱星33碎片云的全数据集中检验了所提算法有效性。与相关MCTS变体方法以及贪婪启发算法对比，所提方法能在测试数据集上更高效地获得较优规划结果，较好地平衡了探索与利用。

关键词: 空间碎片清除, 任务规划, 强化学习, 启发算子, 蒙特卡罗树搜索

Abstract: Vigorous development of the space industry leads to a nonnegligible space debris threat to future space activities. The Active multi-Debris Removal (ADR) technology has become an indispensable means to alleviate this situation. Aiming at the large-scale multi-debris active removal mission planning problem, a Reinforcement Learning (RL) planning scheme is first proposed based on the maximal-reward optimization model for the ADR problem, and the state, action, and reward function of this problem are defined according to the RL framework. Based on an efficient heuristics method, a specialized Monte Carlo Tree Search (MCTS) algorithm is then presented, with the Monte Carlo Tree Search as the core structure and efficient heuristic operators and reinforcement learning iteration process. Finally, its effectiveness is tested in the large-scale complete Iridium 33 debris cloud. The results show that this method is superior to the original MCTS algorithm and the heuristic greedy algorithm.

Key words: active debris removal, mission planning, reinforcement learning, heuristic operator, Monte Carlo tree search

中图分类号:

V448.2

杨家男, 侯晓磊, HU Yu Hen, 刘勇, 潘泉, 冯乾. 基于启发强化学习的大规模ADR任务优化方法[J]. 航空学报, 2021, 42(4): 524354-524354.

YANG Jianan, HOU Xiaolei, HU Yu Hen, LIU Yong, PAN Quan, FENG Qian. Heuristic enhanced reinforcement learning method for large-scale multi-debris active removal mission planning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524354-524354.

参考文献

[1] LIOU J C, JOHNSON N L. Risks in space from orbiting debris[J]. Science, 2006, 311(5759):340-341.
[2] LIOU J C, SHOOTS D. Space missions and satellite box score[J]. Orbital Debris Quarterly News, 2014, 18(2):10.
[3] LIOU J C. Engineering and technology challenges for active debris removal[J]. Progress in Propulsion Physics, 2013, 4:735-748.
[4] KESSLER D J, JOHNSON N L, LIOU J, et al. The Kessler syndrome:Implications to future space operations[J]. Advances in the Astronautical Sciences, 2010, 137(8):2010.
[5] LIOU J C, KRISKO P. An update on the effectiveness of Postmission disposal in LEO[C]//64th International Astronautical Congress. Amsterdam:Elsevier, 2013:23-27.
[6] LIOU J C. An active debris removal parametric study for LEO environment remediation[J]. Advances in Space Research, 2011, 47(11):1865-1876.
[7] 曹喜滨, 李峰, 张锦绣, 等. 空间碎片天基主动清除技术发展现状及趋势[J]. 国防科技大学学报, 2015(4):117-120. CAO X B, LI F, ZHANG J X, et al. Development status and tendency of active debris removal[J]. Journal of National University of Defense Technology, 2015(4):117-120(in Chinese).
[8] 王成林, 张艳, 王鲲鹏. 地基激光清除空间碎片的策略[J]. 北京航空航天大学学报, 2015, 41(11):2137-2143. WANG C L, ZHANG Y, WANG K P. Strategy of removing space debris using ground-based lasers[J]. Journal of Beijing University of Aeronautics and Astronautics, 2015, 41(11):2137-2143(in Chinese).
[9] BRAUN V, L PKEN A, FLEGEL S, et al. Active debris removal of multiple priority targets[J]. Advances in Space Research, 2013, 51(9):1638-1648.
[10] BEREND N, OLIVE X. Bi-objective optimization of a multiple-target active debris removal mission[J]. Acta Astronautica, 2016, 122:324-335.
[11] LIU Y, YANG J. A multi-objective planning method for multi-debris active removal mission in LEO[C]//AIAA Guidance, Navigation, and Control Conference. Reston:AIAA, 2017:1733.
[12] LIU Y, YANG J, WANG Y, et al. Multi-objective optimal preliminary planning of multi-debris active removal mission in LEO[J]. Science China Information Sciences, 2017, 60(7):072202.
[13] YANG J, HU Y H, LIU Y, et al. A maximal-reward preliminary planning for multi-debris active removal mission in LEO with a greedy heuristic method[J]. Acta Astronautica, 2018, 149:123-142.
[14] CERF M. Multiple space debris collecting mission:Optimal mission planning[J]. Journal of Optimization Theory and Applications, 2015, 167(1):195-218.
[15] BARBEE B W, ALFANO S, PINON E, et al. Design of spacecraft missions to remove multiple orbital debris objects[C]//IEEE Aerospace Conference. Piacataway:IEEE Press, 2011:1-14.
[16] CERF M. Multiple space debris collecting mission debris selection and trajectory optimization[J]. Journal of Optimization Theory and Applications, 2013, 156(3):761-796.
[17] SHEN H X, ZHANG T J, CASALINO L, et al. Optimization of active debris removal missions with multiple targets[J]. Journal of Spacecraft and Rockets, 2018, 55(1):181-189.
[18] YU J, CHEN X Q, CHEN L H. Optimal planning of LEO active debris removal based on hybrid optimal control theory[J]. Advances in Space Research, 2015, 55(11):2628-2640.
[19] ZUIANI F, VASILE M. Preliminary design of debris removal missions by means of simplified models for low-thrust, many-revolution transfers[J]. International Journal of Aerospace Engineering, 2012:836250.
[20] YU J, CHEN X Q, CHEN L H, et al. Optimal scheduling of GEO debris removing based on hybrid optimal control theory[J]. Acta Astronautica, 2014, 93(1):400-409.
[21] MADAKAT D, MORIO J, VANDERPOOTEN D. Biobjective planning of an active debris removal mission[J]. Acta Astronautica, 2013, 84:182-188.
[22] OLYMPIO J T, FROUVELLE N. Space debris selection and optimal guidance for removal in the SSO with low-thrust propulsion[J]. Acta Astronautica, 2014, 99:263-275.
[23] STUART J, HOWELL K, WILSON R. Application of multi-agent coordination methods to the design of space debris mitigation tours[J]. Advances in Space Research, 2016, 57(8):1680-1697.
[24] IZZO D, GETZNER I, HENNES D, et al. Evolving solutions to tsp variants for active space debris removal[C]//Annual Conference on Genetic and Evolutionary Computation. New York:ACM, 2015:1207-1214.
[25] MERETA A, IZZO D, WITTIG A. Machine learning of optimal low-thrust transfers between near-earth objects[C]//International Conference on Hybrid Artificial Intelligence Systems. Asturias:Gijon, 2017:543-553.
[26] IZZO D, TAILOR D, VASILEIOU T. On the stability analysis of optimal state feedbacks as represented by deep neural models[DB/OL]. arXiv preprint:1812.02532,2018.
[27] 刘冰雁, 叶雄兵, 周赤非, 等. 基于改进DQN的复合模式在轨服务资源分配[J]. 航空学报, 2020, 41(5):323630. LIU B Y, YE X B, ZHOU C F, et al. Allocation of composite mode on-orbit service resource based on improved DQN[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(5):323630(in Chinese).
[28] 左家亮, 杨任农, 张滢, 等. 基于启发式强化学习的空战机动智能决策[J]. 航空学报, 2017, 38(10):321168. ZUO J L, YANG R N, ZHANG Y, et al. Intelligent decision-making in air combat maneuvering based on heuristic reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2017, 38(10):321168(in Chinese).
[29] 相晓嘉, 闫超, 王菖, 等. 基于深度强化学习的固定翼无人机编队协调控制方法[J]. 航空学报, 2021, 42(4):524009 XIANG X J, YAN C, WANG C, et al. Towards coordination control for fixed-wing UAV formation through deep reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(4):524009(in Chinese).
[30] 赵毓, 管公顺, 郭继峰, 等. 基于多智能体强化学习的空间机械臂轨迹规划[J]. 航空学报, 2021, 42(4):524151. ZHAO Y, GUAN G S, GUO J F, et al. Trajectory planning of space manipulator based on multi-agent reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(4):524151(in Chinese).
[31] BIANCHI R A, RIBEIRO C H, COSTA A H. On the relation between ant colony optimization and heuristically accelerated reinforcement learning[C]//1st International Workshop on Hybrid Control of Autonomous System. Pasadena, 2009:49-55.
[32] WANG Z, CHEN C, LI H X, et al. A novel incremental learning scheme for reinforcement learning in dynamic environments[C]//Intelligent Control and Automation. Piscataway:IEEE Press, 2016:2426-2431.
[33] AYDIN M E, FELLOWS R. A reinforcement learning algorithm for building collaboration in multi-agent systems[DB/OL]. arXiv preprint:1711.10574,2017.
[34] KOBER J, BAGNELL J A, PETERS J. Reinforcement learning in robotics:A survey[J]. The International Journal of Robotics Research, 2013, 32(11):1238-1274.
[35] BUSONIU L, BABUSKA R, SCHUTTER B D. A comprehensive survey of multiagent reinforcement learning[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2008, 38(2):156-172.
[36] MUKHOPADHYAY S, TILAK O, CHAKRABARTI S. Reinforcement learning algorithms for uncertain, dynamic, zero-sum games[C]//17th IEEE International Conference on Machine Learning and Applications (ICMLA). Piscataway:IEEE Press, 2018:48-54.
[37] KIUMARSI B, LEWIS F L, MODARES H, et al. Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics[J]. Automatica, 2014, 50(4):1167-1175.
[38] SILVER D, SCHRITTWIESER J, SIMONYAN K, et al. Mastering the game of Go without human knowledge[J]. Nature, 2017, 550(7676):354-359.
[39] ANSELMO L, PARDINI C. Ranking upper stages in low Earth orbit for active removal[J]. Acta Astronautica, 2016, 122:19-27.
[40] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1):1-27. LIU Q, ZHAI J W, ZHANG Z C, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1):1-27(in Chinese).
[41] 唐振韬. 深度强化学习进展:从AlphaGo到AlphaGo Zero[J]. 控制理论与应用, 2017, 34(12):1529-1546. TANG Z T. Recent progress of deep reinforcement learning:From AlphaGo to AlphaGo Zero[J]. Control Theory & Applications, 2017, 34(12):1529-1546(in Chinese).
[42] BROWNE C B, POWLEY E, WHITEHOUSE D, et al. A survey of Monte Carlo tree search methods[J]. IEEE Transactions on Computational Intelligence and AI in Games, 2012, 4(1):1-43.
[43] WATKINS C J C H, DAYAN P. Technical note:Q-learning[J]. Machine Learning, 1992, 8(3-4):279-292.

E-mail：hkxb@buaa.edu.cn

关于我们

期刊社服务

专业学科

封面文章

友情链接

主管单位：中国科学技术协会主办单位：中国航空学会北京航空航天大学

基于启发强化学习的大规模ADR任务优化方法

Heuristic enhanced reinforcement learning method for large-scale multi-debris active removal mission planning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	惠俊鹏, 汪韧, 俞启东. 基于强化学习的再入飞行器“新质”走廊在线生成技术[J]. 航空学报, 2022, 43(9): 325960-325960.
[2]	符小卫, 王辉, 徐哲. 基于DE-MADDPG的多无人机协同追捕策略[J]. 航空学报, 2022, 43(5): 325311-325311.
[3]	罗庆, 张涛, 单鹏, 张文涛, 刘子豪. 基于改进Q学习的IMA系统重构蓝图生成方法[J]. 航空学报, 2021, 42(8): 525792-525792.
[4]	孙智孝, 杨晟琦, 朴海音, 白成超, 葛俊. 未来智能空战发展综述[J]. 航空学报, 2021, 42(8): 525799-525799.
[5]	任峰, 高传强, 唐辉. 机器学习在流动控制领域的应用及发展趋势[J]. 航空学报, 2021, 42(4): 524686-524686.
[6]	李润泽, 张宇飞, 陈海昕. 针对超临界翼型气动修型策略的强化学习[J]. 航空学报, 2021, 42(4): 523810-523810.
[7]	相晓嘉, 闫超, 王菖, 尹栋. 基于深度强化学习的固定翼无人机编队协调控制方法[J]. 航空学报, 2021, 42(4): 524009-524009.
[8]	郭甲, 庞兆君, 岳帅, 杜忠华. 空间绳系组合体的继电型控制离轨策略[J]. 航空学报, 2021, 42(12): 324738-324738.
[9]	陈斌, 王江, 王阳. 战斗机嵌入式训练系统中的智能虚拟陪练[J]. 航空学报, 2020, 41(6): 523467-523467.
[10]	刘冰雁, 叶雄兵, 周赤非, 刘必鎏. 基于改进DQN的复合模式在轨服务资源分配[J]. 航空学报, 2020, 41(5): 323630-323630.
[11]	吴健发, 王宏伦, 黄宇. 大跨时空任务背景下的太阳能无人机任务规划技术研究进展[J]. 航空学报, 2020, 41(3): 623414-623414.
[12]	刘宇轩, 刘虎, 田永亮, 孙聪. 面向林火持续侦察的多无人机分布式控制方法[J]. 航空学报, 2020, 41(2): 323381-323381.
[13]	陈灿, 莫雳, 郑多, 程子恒, 林德福. 非对称机动能力多无人机智能协同攻防对抗[J]. 航空学报, 2020, 41(12): 324152-324152.
[14]	刘冰雁, 叶雄兵, 高勇, 王新波, 倪蕾. 基于分支深度强化学习的非合作目标追逃博弈策略求解[J]. 航空学报, 2020, 41(10): 324040-324040.
[15]	张耀中, 许佳林, 姚康佳, 刘洁凌. 基于DDPG算法的无人机集群追击任务[J]. 航空学报, 2020, 41(10): 324000-324000.