Unmanned Aerial Vehicle （UAV） has been widely adopted to assist Wireless Sensor Networks （WSNs） in performing data collection tasks. However， time window constraints at the sensor nodes pose new challenges. The UAV must not only arrive in the vicinity of each data-transmitting node within its designated time window， but also complete the data collection task before the window closes. Inefficient trajectory planning increases the UAV’s flight distance， which may compromise the completeness of data collection. Although increasing flight speed can shorten travel time， it also accelerates energy consumption， potentially leading to task failure. To address these problems， We formulate a mathematical model for the UAV trajectory planning problem in a time-window-constrained complete data collection scenario， and then propose a reinforcement learning framework based on a Hierarchical Hybrid Action Representation （H-HyAR） to jointly optimize the UAV’s visiting order of target nodes， hovering offset， and flight speed， while capturing the hierarchical dependencies among these factors to minimize the UAV’s flight distance during the data collection task. Experiment results demonstrate that the H-HyAR algorithm outperforms three comparative hybrid action reinforcement learning algorithms and the Proximal Policy Optimization （PPO） algorithm in terms of flight distance and the influencing factors of this metric， while also exhibiting strong robustness and generalization capabilities.