关键词: 无地图导航, 深度强化学习, 确定策梯度, 无人机, 动态环境

Abstract:

To address the challenges of mapping and navigation in unknown dynamic environments for drone navigation systems， a mapless navigation method based on an improved Twin Delayed Deep Deterministic policy gradient （TD3） is proposed. To solve the perception limitations in a mapless environment， the navigation model is defined as a Partially Observable Markov Decision Process （POMDP）. A Gated Recurrent Units （GRU） is introduced to enable the policy network to utilize the temporal information from historical states， allowing it to obtain an optimal policy and avoid falling into local optima. Based on the TD3 algorithm， a softmax operator is employed to the value function， and a dual policy networks is adopted to address issues of policy function instability and value function underestimation in the TD3 algorithm. A non-sparse reward function is designed to resolve the challenge of policy convergence in reinforcement learning under sparse reward conditions. Finally， simulation experiments conducted on the AirSim platform demonstrate that the improved algorithm achieves faster convergence and higher task success rates in drone mapless obstacle avoidance navigation compared to traditional deep reinforcement learning algorithms.

Key words: mapless navigation, deep reinforcement learning, deterministic policy gradient, UAVs, dynamic environment