基于多目标强化学习的太阳能无人机航迹规划

doi:10.7527/S1000-6893.2025.32817

Abstract

Abstract:

There is a significant coupling between the influencing factors of high-altitude long-endurance solar-powered UAVs in harvesting solar energy and gradient wind energy， and optimizing the harvesting efficiency of these two types of energy simultaneously often leads to conflicts. To address this issue， this study proposes a trajectory planning method based on multi-objective reinforcement learning. This method adopts the multi-objective Soft Actor-Critic （SAC） algorithm based on the multi-objective Markov decision process， combines the UAV's energy harvesting power and energy consumption power into a reward vector， and adds randomly generated weights in each update step. The converged trained policy network can output thrust， attack angle， and bank angle commands based on flight information and a given weight vector， enabling the generation of a set of energy-optimal trajectory solutions within the weight space. Simulation results show that compared with the minimum energy consumption strategy and the strategy based on the conventional single-objective SAC algorithm， this method consistently achieves better energy optimization efficiency and can adaptively respond to weight changes of energy objectives. Compared with the offline optimized trajectory solution set based on the Non-dominated Sorting Genetic Algorithm Ⅱ， this method achieves a hypervolume of the trajectory solution set reaching 90.07% of the former while maintaining excellent real-time performance. In addition， this method also demonstrates a certain degree of generalization ability and can adapt to new untrained wind fields.

Key words: solar-powered unmanned aerial vehicle, trajectory planning, dynamic soaring, energy optimization, reinforcement learning, multi-objective reinforcement learning

CLC Number:

V249.122⁺.3

Tichao XU, Wenyue MENG, Jian ZHANG. Trajectory planning of solar powered unmanned aerial vehicles based on multi-objective reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(12): 332817.

Figures/Tables 26

Fig.1

Table 1

Fig.2

Table 2

Table 3

Table 4

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig. 10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Fig.16

Fig.17

Fig.18

Fig.19

Fig.20

Fig.21

Fig.22

References 37

[1]	KLESH A， KABAMBA P. Energy-optimal path planning for solar-powered aircraft in level flight［C］∥AIAA Guidance， Navigation and Control Conference and Exhibit. Reston： AIAA， 2007.
[2]	EDWARDS D J， KAHN A D， KELLY M， et al. Maximizing net power in circular turns for solar and autonomous soaring aircraft［J］. Journal of Aircraft， 2016， 53（5）： 1237-1247.
[3]	AILON A. A path planning approach for unmanned solar-powered aerial vehicles［J］. Renewable Energy Power Quality Journal， 2023， 21： 109-114.
[4]	SPANGELO S C， GILBERT E G. Power optimization of solar-powered aircraft with specified closed ground tracks［J］. Journal of Aircraft， 2013， 50（1）： 232-238.
[5]	HUANG Y， CHEN J G， WANG H L， et al. A method of 3D path planning for solar-powered UAV with fixed target and solar tracking［J］. Aerospace Science and Technology， 2019， 92： 831-838.
[6]	MARTIN R A， GATES N S， NING A， et al. Dynamic optimization of high-altitude solar aircraft trajectories under station-keeping constraints［J］. Journal of Guidance， Control， and Dynamics， 2019， 42（3）： 538-552.
[7]	BOLANDHEMMAT H， THOMSEN B， MARRIOTT J. Energy-optimized trajectory planning for high altitude long endurance （HALE） aircraft［C］∥2019 18th European Control Conference （ECC）. Piscataway： IEEE Press， 2019： 1486-1493.
[8]	SACHS G， LENZ J， HOLZAPFEL F. Unlimited endurance performance of solar UAVs with minimal or zero electrical energy storage［C］∥AIAA Guidance， Navigation， and Control Conference. Reston： AIAA， 2009.
[9]	GAO X Z， HOU Z X， GUO Z， et al. Joint optimization of battery mass and flight trajectory for high-altitude solar-powered aircraft［J］. Proceedings of the Institution of Mechanical Engineers， Part G： Journal of Aerospace Engineering， 2014， 228（13）： 2439-2451.
[10]	王少奇，马东立，杨穆清，等. 高空太阳能无人机三维航迹优化［J］. 北京航空航天大学学报， 2019， 45（5）： 936-943.
	WANG S Q， MA D L， YANG M Q， et al. Three-dimensional optimal path planning for high-altitude solar-powered UAV［J］. Journal of Beijing University of Aeronautics and Astronautics， 2019， 45（5）： 936-943 （in Chinese）.
[11]	MARRIOTT J， TEZEL B， LIU Z， et al. Trajectory optimization of solar-powered high-altitude long endurance aircraft［C］∥2020 6th International Conference on Control， Automation and Robotics （ICCAR）. Piscataway： IEEE Press， 2020： 473-481.
[12]	NI W J， BI Y， WU D， et al. Energy-optimal trajectory planning for solar-powered aircraft using soft actor-critic［J］. Chinese Journal of Aeronautics， 2022， 35（10）： 337-353.
[13]	RICHARDSON P L. Upwind dynamic soaring of albatrosses and UAVs［J］. Progress in Oceanography， 2015， 130： 146-156.
[14]	SACHS G， LESCH K， KNOLL A. Optimal control for maximum energy extraction from wind shear［C］∥Guidance， Navigation and Control Conference. Reston：AIAA， 1989.
[15]	SACHS G. Minimum shear wind strength required for dynamic soaring of albatrosses［J］. Ibis， 2005， 147（1）： 1-10.
[16]	LIU D N， HOU Z X， GAO X Z. Flight modeling and simulation for dynamic soaring with small unmanned air vehicles［J］. Proceedings of the Institution of Mechanical Engineers， Part G： Journal of Aerospace Engineering， 2017， 231（4）： 589-605.
[17]	NIE X Q， ZWENIG A， PIPREK P， et al. Dynamic soaring trajectory optimization considering the path following performance［J］. IEEE Transactions on Aerospace and Electronic Systems， 2025， 61（4）： 9184-9201.
[18]	ZWENIG A， BEYER Y， HONG H C， et al. Trajectory optimization of dynamic soaring considering rigid body and actuator dynamics using warmstarting［C］∥AIAA SCITECH 2025 Forum. Reston： AIAA， 2025.
[19]	ZHONG G X， XI H Z， ZHENG G， et al. The influence of wind shear to the performance of high-altitude solar-powered aircraft［J］. Proceedings of the Institution of Mechanical Engineers， Part G： Journal of Aerospace Engineering， 2014， 228（9）： 1562-1573.
[20]	ZOU W Y， LI N， AN F C， et al. A novel trajectories optimizing method for dynamic soaring based on deep reinforcement learning［J］. Defence Technology， 2025， 46： 99-108.
[21]	PARK S， FANJOY A， GOLUBEV V V. Application of reinforcement learning for autonomous dynamic soaring［C］∥AIAA SCITECH 2025 Forum. Reston： AIAA， 2025.
[22]	BOWER G， FLANZER T， KROO I. Conceptual design of a small UAV for continuous flight over the ocean［C］∥11th AIAA Aviation Technology， Integration， and Operations （ATIO） Conference. Reston： AIAA， 2011.
[23]	刘多能. 固定翼无人机动态滑翔机理与航迹优化研究［D］. 长沙：国防科学技术大学， 2016.
	LIU D N. Research on mechanism and trajectory optimization for dynamic soaring with fixed-wing unmanned aerial vehicles［D］. Changsha： National University of Defense Technology， 2016 （in Chinese）.
[24]	刘思奇，白俊强. 结合动态滑翔技术的小型太阳能无人机飞行能量变化分析［J］. 西北工业大学学报， 2020， 38（1）： 48-57.
	LIU S Q， BAI J Q. Analysis of flight energy variation of small solar UAVs using dynamic soaring technology［J］. Journal of Northwestern Polytechnical University， 2020， 38（1）： 48-57 （in Chinese）.
[25]	张云飞，王宏伦，张梦华，等. 基于强化学习的多能源动态滑翔航迹优化方法［J］. 西北工业大学学报， 2025， 43（1）： 128-139.
	ZHANG Y F， WANG H L， ZHANG M H， et al. Multi energy dynamic soaring trajectory optimization method based on reinforcement learning［J］. Journal of Northwestern Polytechnical University， 2025， 43（1）： 128-139 （in Chinese）.
[26]	YAN Z J， TABASSUM H. Generalized multi-objective reinforcement learning with envelope updates in URLLC-enabled vehicular networks［J］. IEEE Transactions on Vehicular Technology， 2025， 74（11）： 17666-17682.
[27]	LI F Y， MA R C， DU J R， et al. Decomposition-based multi-objective reinforcement learning for dynamic disassembly job shop scheduling with urgency guidance［J］. Swarm and Evolutionary Computation， 2025， 97： 102040.
[28]	KEMPER N， HEIDER M， PIETRUSCHKA D， et al. A comparative study of multi-objective and neuroevolutionary-based reinforcement learning algorithms for optimizing electric vehicle charging and load management［J］. Applied Energy， 2025， 391： 125890.
[29]	SUN G， XIAO J， LI J H， et al. Aerial reliable collaborative communications for terrestrial mobile users via evolutionary multi-objective deep reinforcement learning［J］. IEEE Transactions on Mobile Computing， 2025， 24（7）： 5731-5748.
[30]	CLARK C A， ALBARADO K M， WILSON J P， et al. Explainability for unmanned aerial vehicle control via multi-objective reinforcement learning［C］∥2025 IEEE Aerospace Conference. Piscataway： IEEE Press， 2025.
[31]	XI Z Y， WU D， NI W J， et al. Energy-optimized trajectory planning for solar-powered aircraft in a wind field using reinforcement learning［J］. IEEE Access， 2022， 10： 87715-87732.
[32]	ETKIN B. Dynamics of atmospheric flight： Chelmsford［M］. North Chelmsford： Courier Corporation， 2012：149-150.
[33]	KEIDEL B. Design and simulation of high-altitude long-endurance solar-powered drones solardrohnen［D］. Munich： Technical University of Munich， 2000 （in Genman）.
[34]	LU H Y， HERMAN D， YU Y L. Multi-objective reinforcement learning： Convexity， stationarity and Pareto optimality［C］∥International Conference on Learning Representations， 2023.
[35]	刘思奇，白俊强. 基于六自由度模型的高空动态滑翔探究［J］. 西北工业大学学报， 2021， 39（4）： 703-711.
	LIU S Q， BAI J Q. Exploration of high-altitude dynamic soaring based on six-degree-of-freedom model［J］. Journal of Northwestern Polytechnical University， 2021， 39（4）： 703-711 （in Chinese）.
[36]	PARK S， DEYST J， HOW J. A new nonlinear guidance logic for trajectory tracking［C］∥AIAA Guidance， Navigation， and Control Conference and Exhibit. Reston： AIAA， 2004.
[37]	DEB K， PRATAP A， AGARWAL S， et al. A fast and elitist multiobjective genetic algorithm： NSGA-Ⅱ［J］. IEEE Transactions on Evolutionary Computation， 2002， 6（2）： 182-197.

A_ij	j=1	j=2	j=3
i=1	3.67×10^-1	3.36×10^-3	-1.08×10^-5
i=2	8.18×10^-2	-1.25×10^-5	9.15×10^-6
i=3	3.86×10^-4	-6.12×10^-6	-2.03×10^-6
B_ij	j=1	j=2	j=3
i=1	1.62×10^-2	5.52×10^-4	1.52×10^-5
i=2	5.61×10^-4	1.68×10^-5	1.11×10^-6
i=3	5.98×10^-4	-7.64×10^-6	1.61×10^-7

参数	范围	参数	数值
h/km	15.0~15.5	m/kg	142
V_a/（m·s^-1）	15~80	S/m²	28
α/（°）	-5~10	S_PV/m²	20.4
φ/（°）	-5~5	P_acc/m²	0.4
θ/（°）	-15~15	η_prop	0.82
ψ/（°）	-180~180	η_mot	0.9
T/N	0~100	η_MPPT	0.95
α_s/（°）	-90~90	η_PV	0.9
γ_s/（°）	0~360

动作	取值范围
ΔT/N	［-10， 10］
Δα/（°）	［-5， 5］
Δφ/（°）	［-5， 5］

Trajectory planning of solar powered unmanned aerial vehicles based on multi-objective reinforcement learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 26

References 37

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Yuheng LIU, Li YANG, Qilong HUANG. Optimizing air and missile defense strategies with explainable hierarchical reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(8): 332786-332786.
[2]	Hao ZHANG, Jianing LIU, Zhi XU, Yuanxin YANG. Trajectory prediction method of incoming missiles based on improved inverse reinforcement learning in aircraft active defense mode [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(8): 332753-332753.
[3]	Wei XIONG, Dong ZHANG, Shuheng YANG, Zhi REN, Wenyi LIU. Manned/unmanned aerial vehicle collaborative interpretable method for intelligent air combat [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(7): 332547-332547.
[4]	Yanlong HAN, An ZHANG, Wenhao BI, Qiucen FAN, Tianle HOU. Intelligent decision-making of airborne terminal infrared composite jamming based on DACTM-PPO [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(7): 332759-332759.
[5]	Sihua GAO, Bingyang ZHAO, Jianfu LI. UAV complete data collection trajectory planning algorithm based on time window constraints [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(6): 332451-332451.
[6]	Yunxiao LIAN, Ni LI, Feng XIE, Pan ZHOU, Changyin DONG. A multi-UAV cooperative air combat decision-making method based on spatial-temporal information fusion [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(6): 332633-332633.
[7]	Yao ZHAO, Xi ZHANG, Di ZHOU, Yutang LI, Siyuan LI. Rapid reentry trajectory planning for hypersonic vehicles with proactive no-fly zone separation assurance [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(6): 332509-332509.
[8]	Lei ZHANG, Can TIAN, Fangqing WEN, Qinghe ZHANG, Han LIU. Multi-objective evolution with deep deterministic strategy gradient algorithm for mobile edge networks [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(3): 631880-631880.
[9]	Wenlin LIU, Xikun HU, Ping ZHONG. Reinforcement learning-driven object detection method for degraded remote sensing images [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(10): 532861-532861.
[10]	Zan MA, Jie BAI, Liqin YAN, Yong CHEN, Shuguang SUN. Safety assessment for airborne intelligent avoidance system based on Bayesian optimization [J]. Acta Aeronautica et Astronautica Sinica, 2026, 47(1): 331973-331973.
[11]	Tao ZHANG, Pan LI, Zixu WANG, Zhenhua ZHU. Design of reward functions for helicopter attitude control in reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(S1): 732184-732184.
[12]	Kaifang WAN, Zhilin WU, Yunhui WU, Haozhi QIANG, Yibo WU, Bo LI. Cooperative location of multiple UAVs with deep reinforcement learning in GPS-denied environment [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 331024-331024.
[13]	Lingfeng JIANG, Xinkai LI, Hai ZHANG, Hanwei LI, Hongli ZHANG. Mapless navigation of UAVs in dynamic environments based on an improved TD3 algorithm [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 331035-331035.
[14]	Zhuoran ZHANG, Jian ZHANG, Guangyuan HU, Han XUE, Hanqi LI, Li YU. Thermal management technologies of high-power-density high-efficiency electric machine systems for more electric aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(6): 531380-531380.
[15]	Zhengyu SONG. Promoting continuous innovation in space transportation systems: Control technologies and challenges [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(6): 531446-531446.