关键词: 深度强化学习, 自主着陆, 抽象SMDP状态转移图, 安全性提升, 实时监控, 反向广度优先搜索

Abstract:

In lunar landing missions， the lander must perform precise operations in extreme environments and often faces the challenge of communication delays. These factors severely limit the real-time operation capabilities of ground control. In response to these challenges， this study proposes a Deep Reinforcement Learning （DRL） framework for safety enhancement based on the Semi-Markov Decision Process （SMDP） to improve the operational safety of autonomous spacecraft landing. To compress the state space and maintain the key characteristics of the decision-making process， this framework compresses the Markov Decision Process （MDP） of the historical trajectory into a SMDP， and constructs an abstract SMDP state transition diagram based on the compressed trajectory. Then， the key state-action pairs of potential risks are identified， and the real-time monitoring and intervention strategy is implemented. The framework effectively improves the safety of the spacecraft’s autonomous landing. Furthermore， the reverse breadth first search method is used to search for the state-action pairs that have decisive impact on task results， and real-time adjustment of the model is realized through the built state-action monitor. Experimental results show that this framework increases the mission success rate of the lunar lander by up to 22% in a simulated environment on the pre-trained Deep Q-Network （DQN）， Dueling DQN， and DDQN models without adding additional sensors or significantly changing the existing system configuration. According to the preset safety evaluation standards， the framework can improve safety by up to 42%. In addition， simulation results in a virtual environment demonstrate the practical application potential of this framework in complex space missions such as lunar landing， which can effectively improve operational safety and efficiency.

Key words: deep reinforcement learning, autonomous landing, abstract SMDP state transition diagram, safety enhancement, real-time monitoring, reverse breadth-first search