基于启发强化学习的大规模ADR任务优化方法

doi:10.7527/S1000-6893.2020.24354

Abstract

Abstract: Vigorous development of the space industry leads to a nonnegligible space debris threat to future space activities. The Active multi-Debris Removal (ADR) technology has become an indispensable means to alleviate this situation. Aiming at the large-scale multi-debris active removal mission planning problem, a Reinforcement Learning (RL) planning scheme is first proposed based on the maximal-reward optimization model for the ADR problem, and the state, action, and reward function of this problem are defined according to the RL framework. Based on an efficient heuristics method, a specialized Monte Carlo Tree Search (MCTS) algorithm is then presented, with the Monte Carlo Tree Search as the core structure and efficient heuristic operators and reinforcement learning iteration process. Finally, its effectiveness is tested in the large-scale complete Iridium 33 debris cloud. The results show that this method is superior to the original MCTS algorithm and the heuristic greedy algorithm.

Key words: active debris removal, mission planning, reinforcement learning, heuristic operator, Monte Carlo tree search

CLC Number:

V448.2

YANG Jianan, HOU Xiaolei, HU Yu Hen, LIU Yong, PAN Quan, FENG Qian. Heuristic enhanced reinforcement learning method for large-scale multi-debris active removal mission planning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524354-524354.

References

[1] LIOU J C, JOHNSON N L. Risks in space from orbiting debris[J]. Science, 2006, 311(5759):340-341.
[2] LIOU J C, SHOOTS D. Space missions and satellite box score[J]. Orbital Debris Quarterly News, 2014, 18(2):10.
[3] LIOU J C. Engineering and technology challenges for active debris removal[J]. Progress in Propulsion Physics, 2013, 4:735-748.
[4] KESSLER D J, JOHNSON N L, LIOU J, et al. The Kessler syndrome:Implications to future space operations[J]. Advances in the Astronautical Sciences, 2010, 137(8):2010.
[5] LIOU J C, KRISKO P. An update on the effectiveness of Postmission disposal in LEO[C]//64th International Astronautical Congress. Amsterdam:Elsevier, 2013:23-27.
[6] LIOU J C. An active debris removal parametric study for LEO environment remediation[J]. Advances in Space Research, 2011, 47(11):1865-1876.
[7] 曹喜滨, 李峰, 张锦绣, 等. 空间碎片天基主动清除技术发展现状及趋势[J]. 国防科技大学学报, 2015(4):117-120. CAO X B, LI F, ZHANG J X, et al. Development status and tendency of active debris removal[J]. Journal of National University of Defense Technology, 2015(4):117-120(in Chinese).
[8] 王成林, 张艳, 王鲲鹏. 地基激光清除空间碎片的策略[J]. 北京航空航天大学学报, 2015, 41(11):2137-2143. WANG C L, ZHANG Y, WANG K P. Strategy of removing space debris using ground-based lasers[J]. Journal of Beijing University of Aeronautics and Astronautics, 2015, 41(11):2137-2143(in Chinese).
[9] BRAUN V, L PKEN A, FLEGEL S, et al. Active debris removal of multiple priority targets[J]. Advances in Space Research, 2013, 51(9):1638-1648.
[10] BEREND N, OLIVE X. Bi-objective optimization of a multiple-target active debris removal mission[J]. Acta Astronautica, 2016, 122:324-335.
[11] LIU Y, YANG J. A multi-objective planning method for multi-debris active removal mission in LEO[C]//AIAA Guidance, Navigation, and Control Conference. Reston:AIAA, 2017:1733.
[12] LIU Y, YANG J, WANG Y, et al. Multi-objective optimal preliminary planning of multi-debris active removal mission in LEO[J]. Science China Information Sciences, 2017, 60(7):072202.
[13] YANG J, HU Y H, LIU Y, et al. A maximal-reward preliminary planning for multi-debris active removal mission in LEO with a greedy heuristic method[J]. Acta Astronautica, 2018, 149:123-142.
[14] CERF M. Multiple space debris collecting mission:Optimal mission planning[J]. Journal of Optimization Theory and Applications, 2015, 167(1):195-218.
[15] BARBEE B W, ALFANO S, PINON E, et al. Design of spacecraft missions to remove multiple orbital debris objects[C]//IEEE Aerospace Conference. Piacataway:IEEE Press, 2011:1-14.
[16] CERF M. Multiple space debris collecting mission debris selection and trajectory optimization[J]. Journal of Optimization Theory and Applications, 2013, 156(3):761-796.
[17] SHEN H X, ZHANG T J, CASALINO L, et al. Optimization of active debris removal missions with multiple targets[J]. Journal of Spacecraft and Rockets, 2018, 55(1):181-189.
[18] YU J, CHEN X Q, CHEN L H. Optimal planning of LEO active debris removal based on hybrid optimal control theory[J]. Advances in Space Research, 2015, 55(11):2628-2640.
[19] ZUIANI F, VASILE M. Preliminary design of debris removal missions by means of simplified models for low-thrust, many-revolution transfers[J]. International Journal of Aerospace Engineering, 2012:836250.
[20] YU J, CHEN X Q, CHEN L H, et al. Optimal scheduling of GEO debris removing based on hybrid optimal control theory[J]. Acta Astronautica, 2014, 93(1):400-409.
[21] MADAKAT D, MORIO J, VANDERPOOTEN D. Biobjective planning of an active debris removal mission[J]. Acta Astronautica, 2013, 84:182-188.
[22] OLYMPIO J T, FROUVELLE N. Space debris selection and optimal guidance for removal in the SSO with low-thrust propulsion[J]. Acta Astronautica, 2014, 99:263-275.
[23] STUART J, HOWELL K, WILSON R. Application of multi-agent coordination methods to the design of space debris mitigation tours[J]. Advances in Space Research, 2016, 57(8):1680-1697.
[24] IZZO D, GETZNER I, HENNES D, et al. Evolving solutions to tsp variants for active space debris removal[C]//Annual Conference on Genetic and Evolutionary Computation. New York:ACM, 2015:1207-1214.
[25] MERETA A, IZZO D, WITTIG A. Machine learning of optimal low-thrust transfers between near-earth objects[C]//International Conference on Hybrid Artificial Intelligence Systems. Asturias:Gijon, 2017:543-553.
[26] IZZO D, TAILOR D, VASILEIOU T. On the stability analysis of optimal state feedbacks as represented by deep neural models[DB/OL]. arXiv preprint:1812.02532,2018.
[27] 刘冰雁, 叶雄兵, 周赤非, 等. 基于改进DQN的复合模式在轨服务资源分配[J]. 航空学报, 2020, 41(5):323630. LIU B Y, YE X B, ZHOU C F, et al. Allocation of composite mode on-orbit service resource based on improved DQN[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(5):323630(in Chinese).
[28] 左家亮, 杨任农, 张滢, 等. 基于启发式强化学习的空战机动智能决策[J]. 航空学报, 2017, 38(10):321168. ZUO J L, YANG R N, ZHANG Y, et al. Intelligent decision-making in air combat maneuvering based on heuristic reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2017, 38(10):321168(in Chinese).
[29] 相晓嘉, 闫超, 王菖, 等. 基于深度强化学习的固定翼无人机编队协调控制方法[J]. 航空学报, 2021, 42(4):524009 XIANG X J, YAN C, WANG C, et al. Towards coordination control for fixed-wing UAV formation through deep reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(4):524009(in Chinese).
[30] 赵毓, 管公顺, 郭继峰, 等. 基于多智能体强化学习的空间机械臂轨迹规划[J]. 航空学报, 2021, 42(4):524151. ZHAO Y, GUAN G S, GUO J F, et al. Trajectory planning of space manipulator based on multi-agent reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(4):524151(in Chinese).
[31] BIANCHI R A, RIBEIRO C H, COSTA A H. On the relation between ant colony optimization and heuristically accelerated reinforcement learning[C]//1st International Workshop on Hybrid Control of Autonomous System. Pasadena, 2009:49-55.
[32] WANG Z, CHEN C, LI H X, et al. A novel incremental learning scheme for reinforcement learning in dynamic environments[C]//Intelligent Control and Automation. Piscataway:IEEE Press, 2016:2426-2431.
[33] AYDIN M E, FELLOWS R. A reinforcement learning algorithm for building collaboration in multi-agent systems[DB/OL]. arXiv preprint:1711.10574,2017.
[34] KOBER J, BAGNELL J A, PETERS J. Reinforcement learning in robotics:A survey[J]. The International Journal of Robotics Research, 2013, 32(11):1238-1274.
[35] BUSONIU L, BABUSKA R, SCHUTTER B D. A comprehensive survey of multiagent reinforcement learning[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2008, 38(2):156-172.
[36] MUKHOPADHYAY S, TILAK O, CHAKRABARTI S. Reinforcement learning algorithms for uncertain, dynamic, zero-sum games[C]//17th IEEE International Conference on Machine Learning and Applications (ICMLA). Piscataway:IEEE Press, 2018:48-54.
[37] KIUMARSI B, LEWIS F L, MODARES H, et al. Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics[J]. Automatica, 2014, 50(4):1167-1175.
[38] SILVER D, SCHRITTWIESER J, SIMONYAN K, et al. Mastering the game of Go without human knowledge[J]. Nature, 2017, 550(7676):354-359.
[39] ANSELMO L, PARDINI C. Ranking upper stages in low Earth orbit for active removal[J]. Acta Astronautica, 2016, 122:19-27.
[40] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1):1-27. LIU Q, ZHAI J W, ZHANG Z C, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1):1-27(in Chinese).
[41] 唐振韬. 深度强化学习进展:从AlphaGo到AlphaGo Zero[J]. 控制理论与应用, 2017, 34(12):1529-1546. TANG Z T. Recent progress of deep reinforcement learning:From AlphaGo to AlphaGo Zero[J]. Control Theory & Applications, 2017, 34(12):1529-1546(in Chinese).
[42] BROWNE C B, POWLEY E, WHITEHOUSE D, et al. A survey of Monte Carlo tree search methods[J]. IEEE Transactions on Computational Intelligence and AI in Games, 2012, 4(1):1-43.
[43] WATKINS C J C H, DAYAN P. Technical note:Q-learning[J]. Machine Learning, 1992, 8(3-4):279-292.

Heuristic enhanced reinforcement learning method for large-scale multi-debris active removal mission planning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Honglin ZHANG, Jianjun LUO, Weihua MA. Spacecraft game decision making for threat avoidance of space targets based on machine learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(8): 329136-329136.
[2]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.
[3]	Shengzhe SHAN, Weiwei ZHANG. Air combat intelligent decision-making method based on self-play and deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328723-328723.
[4]	Jiaxiu YANG, Xinkai LI, Hongli ZHANG, Hao WANG. Time-varying formation control for heterogeneous clusters with switching topologies via reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(10): 329166-329166.
[5]	Bing XIAO, Haichao ZHANG. Reinforcement learning robust optimal control for spacecraft attitude stabilization [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(1): 628890-628890.
[6]	Xuejian WANG, Yongming WEN, Xiaorong SHI, Ningning ZHANG, Jiexi LIU. Design of hybrid intelligent decision framework for multi⁃agent and multi⁃coupling tasks [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729770-729770.
[7]	Youpeng DENG, Jiaxuan FAN, Yan ZHENG, Zhenya WANG, Yongliang LYU, Yuxiao LI. Multiagent opponent modeling with incompleted information [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729782-729782.
[8]	Weilin NI, Yonghai WANG, Cong XU, Fenghua CHI, Haizhao LIANG. Cooperative game guidance method for hypersonic vehicles based on reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729400-729400.
[9]	Zhilin FAN, Hongyong YANG, Yilin HAN. Target round-up control for multi-agent systems based on reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S1): 727487-727487.
[10]	Xiaowei FU, Zhe XU, Jindong ZHU, Nan WANG. Maneuvering decision-making of multi-UAV attack-defence confrontation based on PER-MATD3 [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(7): 327083-327083.
[11]	Xizhen GAO, Liang TANG, Huang HUANG. Deep reinforcement learning in autonomous manipulation for celestial bodies exploration: Applications and challenges [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(6): 26762-026762.
[12]	Pan ZHOU, Jiangtao HUANG, Sheng ZHANG, Gang LIU, Bowen SHU, Jigang TANG. Intelligent air combat decision making and simulation based on deep reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(4): 126731-126731.
[13]	Yupeng FU, Xiangyang DENG, Ziqiang ZHU, Limin ZHANG. Value-filter based air-combat maneuvering optimization [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(22): 628871-628871.
[14]	Chenglei YUE, Xuechuan WANG, Xiaokui YUE, Ting SONG. A spacecraft rendezvous and docking method based on inverse reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 328420-328420.
[15]	Xiangwei ZHU, Dan SHEN, Kai XIAO, Yuexin MA, Xiang LIAO, Fuqiang GU, Fangwen YU, Kefu GAO, Jingnan LIU. Mechanisms, algorithms, implementation and perspectives of brain⁃inspired navigation [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 28569-028569.