局部引导强化学习的舰载机自主调运方法

doi:10.7527/S1000-6893.2024.31333

Abstract

Abstract:

The limited deck space and highly dynamic environment pose significant challenges for autonomous dispatching of carrier-based aircraft. While existing reinforcement learning-based automatic parking techniques offer novel technological insights for autonomous carrier aircraft dispatching， these methods encounter non-convergence issues when directly applied to dynamic environments with constrained aircraft postures. To address this limitation， this paper proposes a locally guided reinforcement learning approach for carrier aircraft autonomous dispatching. The method introduces dual reward mechanisms： a reference trajectory-based local target state reward and a local state grid reward near the dispatching endpoint. These mechanisms effectively guide the learning process， preventing both local optima entrapment and convergence failure during training， thereby significantly enhancing the success rate of autonomous carrier aircraft dispatching. Experimental results demonstrate that the proposed approach outperforms conventional autonomous dispatching methods in terms of both success rate and operational safety. The method’‍s effectiveness has been validated in various mission scenarios and different carrier aircraft configurations.

Key words: carrier-based aircraft, autonomous dispatching, deep reinforcement learning, local target state, local state grid

CLC Number:

Zheng WANG, Hua WANG, Keke CUI, Chaochao LI, Junnan LIU, Mingliang XU. Locally guided reinforcement learning for autonomous dispatching of carrier-based aircraft[J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531333.

Figures/Tables 15

Fig.1

Fig.2

Fig.3

Table 1

Event rewards in reinforcement learning training process

奖励参数	数值	奖励参数	数值
$R e n d d$	1	$R v$	-0.000 15
$R e n d \| v \|$	0.5	$R o$	-1
$R e n d θ$	0.5	$R l$	-0.000 3
$R l o c d$	0.01	$R f l o c$	-0.000 25
$R l o c \| v \|$	0.005	$R u$	-0.002
$R l o c θ$	0.005	$R c l o c$	-0.000 75

Table 1

Fig.4

Table 2

Hyperparameter values of the proposed method

参数	数值	描述
$F m a x$ /N	30 000	最大驱动力
$m$ /kg	15 000	舰载机质量
$β m a x$ /（°）	30	最大转向角
$v m a x$ /（m·s^-1）	5	最大调运速度
$w$ /m	31	局部状态网格宽度
$r$	15	单元格数
$N u m$	24	单元格中的角度步长
$T / s$	0.004	模拟步骤
学习率	3×10^-4	用于梯度下降更新
$γ$	0.99	折扣因子
$H$	15 000	每段最大步数
Epochs	3	训练轮数
Batch Size	1 024	批大小
缓冲区大小	10 240	缓冲区大小
$β$	5×10^-3	熵正则化强度
$ϵ$	0.2	偏差阈值

Table 2

Fig.5

Table 3

Table 4

Fig.6

Fig.7

Fig.8

Table 5

Table 6

Dispatching simulation results under different numbers of local state grids

$w$	碰撞次数	成功率/%
$11$	2	62.50
$21$	0	75.00
$31$	0	87.50
$41$	0	87.50

Table 6

Table 7

Dispatching simulation results under different angle step sizes

$N u m$	碰撞次数	成功率/%
6	3	50.00
12	0	75.00
24	0	87.50
36	0	87.50

Table 7

References 23

[1]	WANG X W， LIU J， SU X C， et al. A review on carrier aircraft dispatch path planning and control on deck‍［J］. Chinese Journal of Aeronautics， 2020， 33（12）： 3039-3057.
[2]	刘洁，韩维，徐卫国，等. 基于滚动时域的舰载机甲板运动轨迹跟踪最优控制［J］. 航空学报‍， 2019， 40（8）： 175-200.
	LIU J， HAN W， XU W G， et al. Optimal path tracking control of carrier-based aircraft on the deck based on RHC‍［J］. Acta Aeronautica et Astronautica Sinica， 2019， 40（8）： 175-200 （in Chinese）.
[3]	LI Y T， WU Y， SU X C， et al. Path planning for aircraft fleet launching on the flight deck of carriers［J］. Mathematics， 2018， 6（10）： 175.
[4]	SU X C， LI Z Y， SONG J Y， et al. A path planning method for carrier aircraft on deck combining artificial experience and intelligent search［J］. IOP Conference Series： Materials Science and Engineering， 2018， 381： 012194.
[5]	ZHANG P Z， XIONG L， YU Z P， et al. Reinforcement learning-based end-to-end parking for automatic parking system［J］. Sensors， 2019， 19（18）： 3996.
[6]	SONG S Y， CHEN H， SUN H W， et al. Data efficient reinforcement learning for integrated lateral planning and control in automated parking system［J］. Sensors， 2020， 20（24）： 7297.
[7]	CHEN S Y， WANG M L， YANG Y， et al. Conflict-constrained multi-agent reinforcement learning method for parking trajectory planning［C］‍∥2023 IEEE International Conference on Robotics and Automation （ICRA）. Piscataway： IEEE Press， 2023.
[8]	CHAI R Q， TSOURDOS A， SAVVARIS A， et al. Design and implementation of deep neural network-based control for automatic parking maneuver process‍［J］. IEEE Transactions on Neural Networks and Learning Systems， 2022， 33（4）： 1400-1413.
[9]	CHAI R Q， LIU D R， LIU T H， et al. Deep learning-based trajectory planning and control for autonomous ground vehicle parking maneuver［J］. IEEE Transactions on Automation Science and Engineering， 2023， 20（3）： 1633-1647.
[10]	张智，林圣琳，朱齐丹，等. 考虑运动学约束的不规则目标遗传避碰规划算法［J］. 航空学报， 2015， 36（4）： 1348-1358.
	ZHANG Z， LIN S L， ZHU Q D， et al. Genetic collision avoidance planning algorithm for irregular shaped object with kinematics constraint［J］. Acta Aeronautica et Astronautica Sinica， 2015， 36（4）： 1348-1358 （in Chinese）.
[11]	ZHANG J， YU J， QU X J， et al. Path planning for carrier aircraft based on geometry and Dijkstra’s algorithm［C］‍∥2017 3rd IEEE International Conference on Control Science and Systems Engineering （ICCSSE）. Piscataway： IEEE Press， 2017.
[12]	WU Y， QU X J. Obstacle avoidance and path planning for carrier aircraft launching［J］. Chinese Journal of Aeronautics， 2015， 28（3）： 695-703.
[13]	薛均晓，孔祥燕，郭毅博，等. 基于深度强化学习的舰载机动态避障方法［J］. 计算机辅助设计与图形学学报， 2021， 33（7）： 1102-1112.
	XUE J X， KONG X Y， GUO Y B， et al. Dynamic obstacle avoidance method for carrier aircraft based on deep reinforcement learning‍［J］. Journal of Computer-Aided Design & Computer Graphics， 2021， 33（7）： 1102-1112 （in Chinese）.
[14]	SHANG Z H， MAO Z Q， ZHANG H C， et al. Collaborative path planning of multiple carrier-based aircraft based on multi-agent reinforcement learning［C］‍∥2022 23rd IEEE International Conference on Mobile Data Management （MDM）. Piscataway： IEEE Press， 2022.
[15]	LIU J， DONG X Z， WANG X W， et al. A homogenization-planning-tracking method to solve cooperative autonomous motion control for heterogeneous carrier dispatch systems‍［J］. Chinese Journal of Aeronautics， 2022， 35（9）： 293-305.
[16]	WANG X W， PENG H J， LIU J， et al. Optimal control based coordinated taxiing path planning and tracking for multiple carrier aircraft on flight deck‍［J］. Defence Technology， 2022， 18（2）： 238-248.
[17]	LIU J， HAN W， WANG X W， et al. Research on cooperative trajectory planning and tracking problem for multiple carrier aircraft on the deck［J］. IEEE Systems Journal， 2020， 14（2）： 3027-3038.
[18]	LIU J， HAN W， LIU C， et al. A new method for the optimal control problem of path planning for unmanned ground systems‍［J］. IEEE Access， 2018， 6： 33251-33260.
[19]	SUTTON R S， BARTO A G. Reinforcement learning： An introduction‍［M］. Cambridge： MIT Press， 2018： 53-80.
[20]	DOLGOV D， THRUN S， MONTEMERLO M， et al. Practical search techniques in path planning for autonomous driving［C］‍∥AAAI Workshop-Technical Report. Menlo Park： AAAI， 2008.
[21]	SCHULMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms‍［DB/OL］. arXiv preprint： 1707. 06347， 2017.
[22]	BENGIO Y， LOURADOUR J， COLLOBERT R， et al. Curriculum learning［C］‍∥Proceedings of the 26th Annual International Conference on Machine Learning. New York： ACM， 2009.
[23]	JULIANI A， BERGES V P， TENG E， et al. Unity： A general platform for intelligent agents［DB/OL］. arXiv preprint： 1809. 02627， 2018.

起始状态		终点状态		起始时间1/s	起始时间2/s	起始时间3/s	起始时间4/s
坐标	角度/（°）	坐标	角度/（°）	起始时间1/s	起始时间2/s	起始时间3/s	起始时间4/s
（323.6，133.1）	334.1	（229.1，129.65）	264.3	3	5	4	5
（217.9，164.9）	202.3	（124.1，163.8）	265.5	6	0	3	6
（302.5，130.7）	340.2	（241.8，116.2）	269.1	1	1	2	6
（241.2，167.9）	192.0	（229.1，129.65）	264.3	2	0	1	1
（275.1，115.4）	0.8	（241.8，116.2）	269.1	6	2	5	6
（190.7，177.9）	178.4	（229.1，129.65）	264.3	6	0	1	1
（142.8，178.7）	178.4	（130.5，143.9）	267.5	6	3	2	2
（337.3，159.8）	257.7	（124.1，163.8）	265.5	6	2	1	4

方法	碰撞次数	成功率/%
Dubins-RVO	2	62.50
Dubins-RVO-PID	4	50.00
HPT	5	59.38
AVP	3	56.25
本文方法	0	87.50

Locally guided reinforcement learning for autonomous dispatching of carrier-based aircraft

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 23

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Kaifang WAN, Zhilin WU, Yunhui WU, Haozhi QIANG, Yibo WU, Bo LI. Cooperative location of multiple UAVs with deep reinforcement learning in GPS-denied environment [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 331024-331024.
[2]	Lingfeng JIANG, Xinkai LI, Hai ZHANG, Hanwei LI, Hongli ZHANG. Mapless navigation of UAVs in dynamic environments based on an improved TD3 algorithm [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(8): 331035-331035.
[3]	Min YANG, Guanjun LIU, Ziyuan ZHOU. Control of lunar landers based on secure reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(3): 630553-630553.
[4]	Xiaochen LYU, Jingping SHI, Yongxi LYU, Gengnong LI. Flow angle reconstruction algorithm for MAGIC CARPET landing with sensor failure [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531159-531159.
[5]	Fang GUO, Wei HAN, Yujie LIU, Jie LIU, Xichao SU, Liangliang CHENG. Scheduling for maintenance and service support of carrier-based aircraft based on variable operation process [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531195-531195.
[6]	Yu LI, Tongwen CHEN, Zhigang WANG, Chiyung WEN, Xiaoxiong LIU. Incremental control of direct lift landing based on predefined-time theory [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531163-531163.
[7]	Wei CHEN, Lulu LI, Dong CHEN, Shaohui ZHANG, Yafei LI, Ke WANG, Yuanyuan JIN, Mingliang XU. Multi-aircraft cooperative decision-making methods driven by differentiated support demands for carrier-based aircraft [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 331274-331274.
[8]	Xudong CHEN, Qiqi CHEN, Yizhe LUO, Jiabao WANG, Mingliang XU. Dynamic parallel scheduling of heterogeneous carrier-based aircraft deck support operations [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(13): 531329-531329.
[9]	Wenhui LING, Chunhui MU, Lingcong NIE, Xian DU, Ximing SUN. Improved DDPG-based multipoint pressure distribution control of variable geometry scramjet combustor at wide range velocities [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(12): 131092-131092.
[10]	Zijie YU, Zheng ZHENG, Qingdong LI, Lin GUO, Suping REN, Jian GUO. Trajectory planning for solar-powered UAVs based on deep reinforcement learning [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(12): 331420-331420.
[11]	Shuyi GAO, Defu LIN, Duo ZHENG, Cheng XU. Intelligent maneuvering penetration guidance strategies for aerial vehicles considering interceptor detection capability limitations [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(10): 331304-331304.
[12]	Guang LIU, Hua WANG, Youfang LIN, Shuo HE, Yafei LI, Mingliang XU. Adaptive batch matching decision method for carrier-based aircraft support operations [J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(1): 330615-330615.
[13]	Dapeng ZHOU, Xiaolei QU. Knowledge-based intelligent pigeon-inspired optimization of carrier-based aircraft landing control [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(S1): 730801-730801.
[14]	Honglin ZHANG, Jianjun LUO, Weihua MA. Spacecraft game decision making for threat avoidance of space targets based on machine learning [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(8): 329136-329136.
[15]	Yunpeng CAI, Dapeng ZHOU, Jiangchuan DING. Intelligent collaborative control of UAV swarms with collision avoidance safety constraints [J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(5): 529683-529683.