基于机器学习的航天器规避目标威胁博弈决策

doi:10.7527/S1000-6893.2023.29136

Abstract

Abstract:

An intelligent decision-making framework and a deep reinforcement learning-based autonomous decision-making method are proposed for the spacecraft decision-making in avoiding the threat of space targets. Taking into account the maneuvering characteristics of space targets and the gameplay of threat avoidance， an intelligent game decision-making framework for spacecraft threat avoidance is proposed based on the Observation-Orientation-Decision-Action （OODA） loop decision-making idea and machine learning techniques. Based on this framework and inference on the motion intentions of space targets， a deep reinforcement learning-based spacecraft maneuver decision-making algorithm and training environment are designed to enable spacecraft decision-making control with game response capability， which realizes the avoidance response to the typical motion intentions of space targets. Furthermore， the generalization of spacecraft autonomous maneuvering decision-making algorithm and its adaptability to possible uncertain maneuvers of space targets are improved by using the self-play learning technique. Finally， the effectiveness of our proposed method is verified through simulations.

Key words: spacecraft maneuver, intelligent decision-making, threat avoidance, OODA loop, deep reinforcement learning

CLC Number:

V448.2

Honglin ZHANG, Jianjun LUO, Weihua MA. Spacecraft game decision making for threat avoidance of space targets based on machine learning[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(8): 329136-329136.

Figures/Tables 21

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Fig.8

Fig.9

Fig.10

Table 2

Settings of simulation hyperparameters

参数	数值
学习率 $α$	0.001
矩估计衰减率	$β 1 = 0.9, β 2 = 0.999$
最大训练回合	4 000
折扣因子	0.99
截断因子 $ε$	0.2
值函数误差系数c₁	0.5
交叉熵系数c₂	0.01

Table 2

Fig.11

Table 3

Initial states of typical intentions of space targets

目标意图	初始状态
定点跟飞	$[0, - 50, - 40, 0,0, 0,0] T$
盘旋跟飞	$[0, - 50, - 40, 0, - 0.1,0.1, 0,0] T$
振荡跟飞	$[0, - 50, - 40, 0,0, 0, - 0.1,0.1] T$
共面绕飞	$[0, - 50, - 40, 0, n y / 2,0, 0] T$
异面绕飞	$[0, - 50, - 40, 0, n y / 2,0, ± 3, n y / 2] T$
飞掠抵近	$[- 21, - 19, - 110, - 90, 0,0, - 3 n x / 2,0] T$
跳跃抵近	$[0, - 55, - 45, 0,0, - 0.2, - 0.1, 0] T$
螺旋抵近	$[0, - 55, - 45, 0, - 0.1,0.1, - 0.2, - 0.1, - 3, 3 v x] T$

Table 3

Fig.12

Fig.13

Fig.14

Table 4

Monte Carlo simulation results of typical intentions

目标意图	指标
目标意图	$T s$ /h	$Δ v$ /（m·s^-1）	$ρ$ /km	$α$ /（°）
定点跟飞	12.5±1.13	1.48±0.15	102.48±1.54	22.13±0.26
盘旋跟飞	11.5±0.70	1.58±0.11	102.52±1.49	15.18±0.12
振荡跟飞	10.5±0.58	1.43±0.10	103.17±1.89	8.62±0.06
共面绕飞	21±3.64	2.51±0.40	102.31±1.96	47.73±0.72
异面绕飞	23.5±2.78	2.72±0.28	102.91±1.95	40.38±0.43
飞掠抵近	12±0.59	1.34±0.10	104.19±2.54	44.30±0.24
跳跃抵近	16.5±1.94	2.04±0.28	104.71±2.70	38.71±0.46
螺旋抵近	16.5±2.60	2.01±0.30	104.77±2.93	42.03±0.47

Table 4

Fig.15

Table 5

Table 6

Comparisons of impulsive velocity consumption

目标意图	$Δ v$ /（m·s^-1）
目标意图	航天器	空间目标
定点跟飞	6.43	6.82
盘旋跟飞	6.92	7.10
振荡跟飞	5.96	6.99
共面绕飞	7.34	8.44
异面绕飞	6.98	7.70
飞掠抵近	2.89	3.15
跳跃抵近	5.91	6.63
螺旋抵近	7.40	7.61

Table 6

References 20

1	袁利，姜甜甜. 航天器威胁规避智能自主控制技术研究综述［J］. 自动化学报， 2023， 49（2）： 229-245.
	YUAN L， JIANG T T. Review on intelligent autonomous control for spacecraft confronting orbital threats［J］. Acta Automatica Sinica， 2023， 49（2）： 229-245 （in Chinese）.
2	袁利. 面向不确定环境的航天器智能自主控制技术［J］. 宇航学报， 2021， 42（7）： 839-849.
	YUAN L. Spacecraft intelligent autonomous control technology toward uncertain environment［J］. Journal of Astronautics， 2021， 42（7）： 839-849 （in Chinese）.
3	王杰，丁达理，董康生，等. UCAV自主空战战术机动动作建模与轨迹生成［J］. 火力与指挥控制， 2018， 43（12）： 42-49.
	WANG J， DING D L， DONG K S， et al. UCAV autonomous air combat tactical maneuvering modeling and trajectory generation［J］. Fire Control & Command Control， 2018， 43（12）： 42-49 （in Chinese）.
4	于大腾，王华，孙福煜. 考虑潜在威胁区的航天器最优规避机动策略［J］. 航空学报， 2017， 38（1）： 320202.
	YU D T， WANG H， SUN F Y. Optimal evasive maneuver strategy with potential threatening area being considered［J］. Acta Aeronautica et Astronautica Sinica， 2017， 38（1）： 320202 （in Chinese）.
5	BOMBARDELLI C. Analytical formulation of impulsive collision avoidance dynamics［J］. Celestial Mechanics and Dynamical Astronomy， 2014， 118（2）： 99-114.
6	GONZALO J L， COLOMBO C， DI LIZIA P. Analytical framework for space debris collision avoidance maneuver design［J］. Journal of Guidance， Control， and Dynamics， 2020， 44（3）： 469-487.
7	BATHER J A， ISAACS R. Differential games： a mathematical theory with applications to warfare and pursuit， control and optimization［J］. Journal of the Royal Statistical Society Series A （General）， 1966， 129（3）： 474.
8	PRINCE E R， HESS J A， COBB R G， et al. Elliptical orbit proximity operations differential games［J］. Journal of Guidance， Control， and Dynamics， 2019， 42（7）： 1458-1472.
9	LIANG L， DENG F， PENG Z H， et al. A differential game for cooperative target defense［J］. Automatica， 2019， 102： 58-71.
10	SUN J L， LIU C S， YE Q. Robust differential game guidance laws design for uncertain interceptor-target engagement via adaptive dynamic programming［J］. International Journal of Control， 2017， 90（5）： 990-1004.
11	WATANABE T， JOHNSON E N. Trajectory generation using deep neural network： AIAA-2018-1893［R］. Reston： AIAA， 2018.
12	IZZO D， TAILOR D， VASILEIOU T. On the stability analysis of deep neural network representations of an optimal state-feedback［DB/OL］. arXiv preprint： 1812.02532， 2018.
13	SÁNCHEZ-SÁNCHEZ C， IZZO D. Real-time optimal control via deep neural networks： Study on landing problems［J］. Journal of Guidance， Control， and Dynamics， 2018， 41（5）： 1122-1135.
14	OESTREICH C E， LINARES R， GONDHALEKAR R. Autonomous six-degree-of-freedom spacecraft docking with rotating targets via reinforcement learning［J］. Journal of Aerospace Information Systems， 2021， 18（7）： 417-428.
15	刘冰雁，叶雄兵，高勇，等. 基于分支深度强化学习的非合作目标追逃博弈策略求解［J］. 航空学报， 2020， 41（10）： 324040.
	LIU B Y， YE X B， GAO Y， et al. Strategy solution of non-cooperative target pursuit-evasion game based on branching deep reinforcement learning［J］. Acta Aeronautica et Astronautica Sinica， 2020， 41（10）： 324040 （in Chinese）.
16	ZHANG J R， ZHANG K P， ZHANG Y， et al. Near-optimal interception strategy for orbital pursuit-evasion using deep reinforcement learning［J］. Acta Astronautica， 2022， 198： 9-25.
17	赵毓，郭继峰，颜鹏，等. 稀疏奖励下多航天器规避决策自学习仿真［J］. 系统仿真学报， 2021， 33（8）： 1766-1774.
	ZHAO Y， GUO J F， YAN P， et al. Self-learning-based multiple spacecraft evasion decision making simulation under sparse reward condition［J］. Journal of System Simulation， 2021， 33（8）： 1766-1774 （in Chinese）.
18	ZHANG H L， LUO J J， GAO Y， et al. An intention inference method for the space non-cooperative target based on BiGRU-Self Attention［J］. Advances in Space Research， 2023， 72（5）： 1815-1828.
19	黎飞，雷拥军，冯佳佳. 一种GEO卫星太阳光遮挡轨迹设计与控制方法［J］. 宇航学报， 2022， 43（2）： 198-205.
	LI F， LEI Y J， FENG J J. A design and control method of Sun occlusion trajectory for GEO satellite［J］. Journal of Astronautics， 2022， 43（2）： 198-205 （in Chinese）.
20	SILVER D， SCHRITTWIESER J， SIMONYAN K， et al. Mastering the game of Go without human knowledge［J］. Nature， 2017， 550： 354-359.

网络层级	策略网络		价值网络
网络层级	单元数	激活函数	单元数	激活函数
输入层	6		6
隐含层1	64	tanh	64	tanh
隐含层2	32	tanh	32	tanh
输出层	3	tanh	1	线性

[1]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.
[2]	Sai ZHANG, Zhen YANG, Xiangnan DU, Yazhong LUO. Threat avoidance strategy of spacecraft maneuvering approach based on orbital reachable domain [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328778-328778.
[3]	Shengzhe SHAN, Weiwei ZHANG. Air combat intelligent decision-making method based on self-play and deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328723-328723.
[4]	Xuejian WANG, Yongming WEN, Xiaorong SHI, Ningning ZHANG, Jiexi LIU. Design of hybrid intelligent decision framework for multi⁃agent and multi⁃coupling tasks [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(S2): 729770-729770.
[5]	Xizhen GAO, Liang TANG, Huang HUANG. Deep reinforcement learning in autonomous manipulation for celestial bodies exploration: Applications and challenges [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(6): 26762-026762.
[6]	Pan ZHOU, Jiangtao HUANG, Sheng ZHANG, Gang LIU, Bowen SHU, Jigang TANG. Intelligent air combat decision making and simulation based on deep reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(4): 126731-126731.
[7]	Xiangwei ZHU, Dan SHEN, Kai XIAO, Yuexin MA, Xiang LIAO, Fuqiang GU, Fangwen YU, Kefu GAO, Jingnan LIU. Mechanisms, algorithms, implementation and perspectives of brain⁃inspired navigation [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(19): 28569-028569.
[8]	Min LI, Li YUAN, Chunling WEI. Spacecraft autonomous fly-around multi-mode control based on hybrid state machine [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(18): 328296-328296.
[9]	Lei DONG, Hongbing CHEN, Xi CHEN, Changxiao ZHAO. Distributed multi-agent coalition task allocation strategy for single pilot operation mode based on DQN [J]. Acta Aeronautica et Astronautica Sinica, 2023, 44(13): 327895-327895.
[10]	Wenxue CHEN, Changsheng GAO, Wuxing JING. Trust region policy optimization guidance algorithm for intercepting maneuvering target [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(11): 327596-327596.
[11]	Sheng ZHANG, Pan ZHOU, Yang HE, Jiangtao HUANG, Gang LIU, Jigang TANG, Huaizhi JIA, Xin DU. Air combat maneuver decision-making test based on deep reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(10): 128094-128094.
[12]	FU Xiaowei, WANG Hui, XU Zhe. Cooperative pursuit strategy for multi-UAVs based on DE-MADDPG algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(5): 325311-325311.
[13]	REN Feng, GAO Chuanqiang, TANG Hui. Machine learning for flow control: Applications and development trends [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524686-524686.
[14]	XIANG Xiaojia, YAN Chao, WANG Chang, YIN Dong. Coordination control method for fixed-wing UAV formation through deep reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524009-524009.
[15]	LIU Bingyan, YE Xiongbing, ZHOU Chifei, LIU Biliu. Allocation of composite mode on-orbit service resource based on improved DQN [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(5): 323630-323630.

Spacecraft game decision making for threat avoidance of space targets based on machine learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 21

References 20

Related Articles 15

Recommended Articles

Metrics

Comments

目标意图	成功率/%
目标意图	固定意图训练	自我博弈训练
定点跟飞	0	100
盘旋跟飞	18.2	100
振荡跟飞	86.1	100
共面绕飞	0	91.8
异面绕飞	26.6	84.1
飞掠抵近	0	97.7
跳跃抵近	0	98.1
螺旋抵近	0	97.7