基于机器学习的航天器规避目标威胁博弈决策

doi:10.7527/S1000-6893.2023.29136

Abstract

Abstract:

An intelligent decision-making framework and a deep reinforcement learning-based autonomous decision-making method are proposed for the spacecraft decision-making in avoiding the threat of space targets. Taking into account the maneuvering characteristics of space targets and the gameplay of threat avoidance， an intelligent game decision-making framework for spacecraft threat avoidance is proposed based on the Observation-Orientation-Decision-Action （OODA） loop decision-making idea and machine learning techniques. Based on this framework and inference on the motion intentions of space targets， a deep reinforcement learning-based spacecraft maneuver decision-making algorithm and training environment are designed to enable spacecraft decision-making control with game response capability， which realizes the avoidance response to the typical motion intentions of space targets. Furthermore， the generalization of spacecraft autonomous maneuvering decision-making algorithm and its adaptability to possible uncertain maneuvers of space targets are improved by using the self-play learning technique. Finally， the effectiveness of our proposed method is verified through simulations.

Key words: spacecraft maneuver, intelligent decision-making, threat avoidance, OODA loop, deep reinforcement learning

CLC Number:

V448.2

Honglin ZHANG, Jianjun LUO, Weihua MA. Spacecraft game decision making for threat avoidance of space targets based on machine learning[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(8): 329136.

Figures/Tables 21

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Fig.8

Fig.9

Fig.10

Table 2

Settings of simulation hyperparameters

参数	数值
学习率 $α$	0.001
矩估计衰减率	$β 1 = 0.9, β 2 = 0.999$
最大训练回合	4 000
折扣因子	0.99
截断因子 $ε$	0.2
值函数误差系数c₁	0.5
交叉熵系数c₂	0.01

Table 2

Fig.11

Table 3

Initial states of typical intentions of space targets

目标意图	初始状态
定点跟飞	$[0, - 50, - 40, 0,0, 0,0] T$
盘旋跟飞	$[0, - 50, - 40, 0, - 0.1,0.1, 0,0] T$
振荡跟飞	$[0, - 50, - 40, 0,0, 0, - 0.1,0.1] T$
共面绕飞	$[0, - 50, - 40, 0, n y / 2,0, 0] T$
异面绕飞	$[0, - 50, - 40, 0, n y / 2,0, ± 3, n y / 2] T$
飞掠抵近	$[- 21, - 19, - 110, - 90, 0,0, - 3 n x / 2,0] T$
跳跃抵近	$[0, - 55, - 45, 0,0, - 0.2, - 0.1, 0] T$
螺旋抵近	$[0, - 55, - 45, 0, - 0.1,0.1, - 0.2, - 0.1, - 3, 3 v x] T$

Table 3

Fig.12

Fig.13

Fig.14

Table 4

Monte Carlo simulation results of typical intentions

目标意图	指标
目标意图	$T s$ /h	$Δ v$ /（m·s^-1）	$ρ$ /km	$α$ /（°）
定点跟飞	12.5±1.13	1.48±0.15	102.48±1.54	22.13±0.26
盘旋跟飞	11.5±0.70	1.58±0.11	102.52±1.49	15.18±0.12
振荡跟飞	10.5±0.58	1.43±0.10	103.17±1.89	8.62±0.06
共面绕飞	21±3.64	2.51±0.40	102.31±1.96	47.73±0.72
异面绕飞	23.5±2.78	2.72±0.28	102.91±1.95	40.38±0.43
飞掠抵近	12±0.59	1.34±0.10	104.19±2.54	44.30±0.24
跳跃抵近	16.5±1.94	2.04±0.28	104.71±2.70	38.71±0.46
螺旋抵近	16.5±2.60	2.01±0.30	104.77±2.93	42.03±0.47

Table 4

Fig.15

Table 5

Table 6

Comparisons of impulsive velocity consumption

目标意图	$Δ v$ /（m·s^-1）
目标意图	航天器	空间目标
定点跟飞	6.43	6.82
盘旋跟飞	6.92	7.10
振荡跟飞	5.96	6.99
共面绕飞	7.34	8.44
异面绕飞	6.98	7.70
飞掠抵近	2.89	3.15
跳跃抵近	5.91	6.63
螺旋抵近	7.40	7.61

Table 6

References 20

1	袁利，姜甜甜. 航天器威胁规避智能自主控制技术研究综述［J］. 自动化学报， 2023， 49（2）： 229-245.
	YUAN L， JIANG T T. Review on intelligent autonomous control for spacecraft confronting orbital threats［J］. Acta Automatica Sinica， 2023， 49（2）： 229-245 （in Chinese）.
2	袁利. 面向不确定环境的航天器智能自主控制技术［J］. 宇航学报， 2021， 42（7）： 839-849.
	YUAN L. Spacecraft intelligent autonomous control technology toward uncertain environment［J］. Journal of Astronautics， 2021， 42（7）： 839-849 （in Chinese）.
3	王杰，丁达理，董康生，等. UCAV自主空战战术机动动作建模与轨迹生成［J］. 火力与指挥控制， 2018， 43（12）： 42-49.
	WANG J， DING D L， DONG K S， et al. UCAV autonomous air combat tactical maneuvering modeling and trajectory generation［J］. Fire Control & Command Control， 2018， 43（12）： 42-49 （in Chinese）.
4	于大腾，王华，孙福煜. 考虑潜在威胁区的航天器最优规避机动策略［J］. 航空学报， 2017， 38（1）： 320202.
	YU D T， WANG H， SUN F Y. Optimal evasive maneuver strategy with potential threatening area being considered［J］. Acta Aeronautica et Astronautica Sinica， 2017， 38（1）： 320202 （in Chinese）.
5	BOMBARDELLI C. Analytical formulation of impulsive collision avoidance dynamics［J］. Celestial Mechanics and Dynamical Astronomy， 2014， 118（2）： 99-114.
6	GONZALO J L， COLOMBO C， DI LIZIA P. Analytical framework for space debris collision avoidance maneuver design［J］. Journal of Guidance， Control， and Dynamics， 2020， 44（3）： 469-487.
7	BATHER J A， ISAACS R. Differential games： a mathematical theory with applications to warfare and pursuit， control and optimization［J］. Journal of the Royal Statistical Society Series A （General）， 1966， 129（3）： 474.
8	PRINCE E R， HESS J A， COBB R G， et al. Elliptical orbit proximity operations differential games［J］. Journal of Guidance， Control， and Dynamics， 2019， 42（7）： 1458-1472.
9	LIANG L， DENG F， PENG Z H， et al. A differential game for cooperative target defense［J］. Automatica， 2019， 102： 58-71.
10	SUN J L， LIU C S， YE Q. Robust differential game guidance laws design for uncertain interceptor-target engagement via adaptive dynamic programming［J］. International Journal of Control， 2017， 90（5）： 990-1004.
11	WATANABE T， JOHNSON E N. Trajectory generation using deep neural network： AIAA-2018-1893［R］. Reston： AIAA， 2018.
12	IZZO D， TAILOR D， VASILEIOU T. On the stability analysis of deep neural network representations of an optimal state-feedback［DB/OL］. arXiv preprint： 1812.02532， 2018.
13	SÁNCHEZ-SÁNCHEZ C， IZZO D. Real-time optimal control via deep neural networks： Study on landing problems［J］. Journal of Guidance， Control， and Dynamics， 2018， 41（5）： 1122-1135.
14	OESTREICH C E， LINARES R， GONDHALEKAR R. Autonomous six-degree-of-freedom spacecraft docking with rotating targets via reinforcement learning［J］. Journal of Aerospace Information Systems， 2021， 18（7）： 417-428.
15	刘冰雁，叶雄兵，高勇，等. 基于分支深度强化学习的非合作目标追逃博弈策略求解［J］. 航空学报， 2020， 41（10）： 324040.
	LIU B Y， YE X B， GAO Y， et al. Strategy solution of non-cooperative target pursuit-evasion game based on branching deep reinforcement learning［J］. Acta Aeronautica et Astronautica Sinica， 2020， 41（10）： 324040 （in Chinese）.
16	ZHANG J R， ZHANG K P， ZHANG Y， et al. Near-optimal interception strategy for orbital pursuit-evasion using deep reinforcement learning［J］. Acta Astronautica， 2022， 198： 9-25.
17	赵毓，郭继峰，颜鹏，等. 稀疏奖励下多航天器规避决策自学习仿真［J］. 系统仿真学报， 2021， 33（8）： 1766-1774.
	ZHAO Y， GUO J F， YAN P， et al. Self-learning-based multiple spacecraft evasion decision making simulation under sparse reward condition［J］. Journal of System Simulation， 2021， 33（8）： 1766-1774 （in Chinese）.
18	ZHANG H L， LUO J J， GAO Y， et al. An intention inference method for the space non-cooperative target based on BiGRU-Self Attention［J］. Advances in Space Research， 2023， 72（5）： 1815-1828.
19	黎飞，雷拥军，冯佳佳. 一种GEO卫星太阳光遮挡轨迹设计与控制方法［J］. 宇航学报， 2022， 43（2）： 198-205.
	LI F， LEI Y J， FENG J J. A design and control method of Sun occlusion trajectory for GEO satellite［J］. Journal of Astronautics， 2022， 43（2）： 198-205 （in Chinese）.
20	SILVER D， SCHRITTWIESER J， SIMONYAN K， et al. Mastering the game of Go without human knowledge［J］. Nature， 2017， 550： 354-359.

网络层级	策略网络		价值网络
网络层级	单元数	激活函数	单元数	激活函数
输入层	6		6
隐含层1	64	tanh	64	tanh
隐含层2	32	tanh	32	tanh
输出层	3	tanh	1	线性

[1]	Kaifang WAN, Zhilin WU, Yunhui WU, Haozhi QIANG, Yibo WU, Bo LI. Cooperative location of multiple UAVs with deep reinforcement learning in GPS-denied environment [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 331024-331024.
[2]	Lingfeng JIANG, Xinkai LI, Hai ZHANG, Hanwei LI, Hongli ZHANG. Mapless navigation of UAVs in dynamic environments based on an improved TD3 algorithm [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 331035-331035.
[3]	Min YANG, Guanjun LIU, Ziyuan ZHOU. Control of lunar landers based on secure reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(3): 630553-630553.
[4]	Chen WANG, Caisheng WEI, Zeyang YIN, Kai JIN, Xingchen LI. Collaborative planning of multi-UAV trajectories and communication strategies considering channel resource constraints [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(18): 331837-331837.
[5]	Yu WANG, Zhipeng XIE, Yongjian TIAN, Guanglei MENG. Distributed UAV formation control with virtual structure guided reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(15): 331354-331354.
[6]	Wei CHEN, Lulu LI, Dong CHEN, Shaohui ZHANG, Yafei LI, Ke WANG, Yuanyuan JIN, Mingliang XU. Multi-aircraft cooperative decision-making methods driven by differentiated support demands for carrier-based aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531274-531274.
[7]	Xudong CHEN, Qiqi CHEN, Yizhe LUO, Jiabao WANG, Mingliang XU. Dynamic parallel scheduling of heterogeneous carrier-based aircraft deck support operations [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531329-531329.
[8]	Zheng WANG, Hua WANG, Keke CUI, Chaochao LI, Junnan LIU, Mingliang XU. Locally guided reinforcement learning for autonomous dispatching of carrier-based aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531333-531333.
[9]	Wenhui LING, Chunhui MU, Lingcong NIE, Xian DU, Ximing SUN. Improved DDPG-based multipoint pressure distribution control of variable geometry scramjet combustor at wide range velocities [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(12): 131092-131092.
[10]	Zijie YU, Zheng ZHENG, Qingdong LI, Lin GUO, Suping REN, Jian GUO. Trajectory planning for solar-powered UAVs based on deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(12): 331420-331420.
[11]	Shuyi GAO, Defu LIN, Duo ZHENG, Cheng XU. Intelligent maneuvering penetration guidance strategies for aerial vehicles considering interceptor detection capability limitations [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(10): 331304-331304.
[12]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.
[13]	Shengzhe SHAN, Weiwei ZHANG. Air combat intelligent decision-making method based on self-play and deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328723-328723.
[14]	Sai ZHANG, Zhen YANG, Xiangnan DU, Yazhong LUO. Threat avoidance strategy of spacecraft maneuvering approach based on orbital reachable domain [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328778-328778.
[15]	Bing GAO, Zhejie ZHANG, Qijie ZOU, Zhiguo LIU, Xiling ZHAO. Multi-agent communication cooperation based on deep reinforcement learning and information theory [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(18): 329862-329862.

Spacecraft game decision making for threat avoidance of space targets based on machine learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 21

References 20

Related Articles 15

Recommended Articles

Metrics

Comments

目标意图	成功率/%
目标意图	固定意图训练	自我博弈训练
定点跟飞	0	100
盘旋跟飞	18.2	100
振荡跟飞	86.1	100
共面绕飞	0	91.8
异面绕飞	26.6	84.1
飞掠抵近	0	97.7
跳跃抵近	0	98.1
螺旋抵近	0	97.7