基于舰面环境模型的舰载机保障作业调度算法
收稿日期: 2025-12-03
修回日期: 2026-03-26
录用日期: 2026-04-28
网络出版日期: 2026-05-14
基金资助
国家自然科学基金(62406292);国家自然科学基金(62302459);国家自然科学基金(62406293);国家自然科学基金(62325602);国家自然科学基金(62036010)
Scheduling algorithm for carrier-based aircraft support operations based on deck environment model
Received date: 2025-12-03
Revised date: 2026-03-26
Accepted date: 2026-04-28
Online published: 2026-05-14
Supported by
National Natural Science Foundation of China(62406292)
在航母舰载机保障作业调度中，无模型强化学习（MFRL）在动态甲板场景下受到物理环境建模精度的制约，而基于模型的强化学习（MBRL）因环境模型与决策模型在迭代训练中存在相互依赖的协同优化问题，面临计算复杂度高与收敛困难的挑战。对此，提出了一种融合有模型与无模型（MB-MF）特性的混合强化学习框架。首先，利用历史调度数据训练基于深度神经网络的甲板环境模型，使其在最小容差范围内精确预测状态转移；然后，将收敛后的环境模型替代真实环境，内嵌入交互环境中，结合深度Q网络（DQN）算法训练调度智能体，实现环境模型学习与策略优化的解耦；最后，经实验验证表明，与使用物理环境的MFRL相比，该方法在无需精确建模的情况下性能差距仅为4%。而相较于MBRL基线方法，舰载机出动时间则缩短34%。同时在资源受限场景中，决策速度较启发式方法提高近300倍，而调度质量仅降低17%。
罗祎喆 , 王佳宝 , 余新得 , 陈旭东 , 金钊 , 冯朔 , 石育澄 , 徐明亮 . 基于舰面环境模型的舰载机保障作业调度算法[J]. 航空学报, 2026 , 47(17) : 333180 -333180 . DOI: 10.7527/S1000-6893.2026.33180
In aircraft carrier flight deck scheduling operations， Model-Free Reinforcement Learning （MFRL） is constrained by the precision of physical environment modeling under dynamic deck conditions. In contrast， Model-Based Reinforcement Learning （MBRL） faces challenges of high computational complexity and convergence difficulties due to the interdependent co-optimization between the environment model and the decision-making model during iterative training. To address these issues， This paper proposes a hybrid reinforcement learning framework that integrates Model-Based and Model-Free （MB-MF） characteristics. First， a deep neural network-based deck environment model is trained using historical scheduling data to accurately predict state transitions within a minimal tolerance range. Then， the converged environment model is embedded into the interactive environment in place of the real environment， where a scheduling agent is trained using the Deep Q-Network （DQN） algorithm， thereby decoupling environment model learning from policy optimization. Experimental results demonstrate that， compared to MFRL using the physical environment， the proposed method achieves a performance gap of only 4% without requiring precise modeling. Moreover， it reduces the aircraft sortie time by 34% relative to the MBRL baseline. In resource-constrained scenarios， the decision-making speed is nearly 300 times faster than that of heuristic methods， while the scheduling quality is reduced by only 17%.
| [1] | DIETZ D C， JENKINS R C. Analysis of aircraft sortie generation with the use of a fork-join queueing network model［J］. Naval Research Logistics， 1997， 44（2）： 153-164. |
| [2] | LIU Y J， HAN W， SU X C， et al. Optimization of fixed aviation support resource station configuration for aircraft carrier based on aircraft dispatch mission scheduling［J］. Chinese Journal of Aeronautics， 2023， 36（2）： 127-138. |
| [3] | WANG X， LIU J， SU X， et al. A review on carrier aircraft dispatch path planning and control on deck［J］. Chinese Journal of Aeronautics， 2020， 33（12）： 3039-3057. |
| [4] | 刘翱， 刘克. 舰载机保障作业调度问题研究进展［J］. 系统工程理论与实践， 2017， 37（1）： 49-60. |
| LIU A， LIU K. Research progress on carrier-based aircraft support operation scheduling problem［J］. Systems Engineering-Theory & Practice， 2017， 37（1）： 49-60 （in Chinese）. | |
| [5] | HARTMANN S， BRISKORN D. An updated survey of variants and extensions of the resource-constrained project scheduling problem［J］. European Journal of Operational Research， 2022， 297（1）： 1-14. |
| [6] | KHATAB A， DIALLO C， VENKATADRI U， et al. Optimization of the joint selective maintenance and repairperson assignment problem under imperfect maintenance［J］. Computers & Industrial Engineering， 2018， 125： 413-422. |
| [7] | 陈小飞， 时立攀， 毕玉泉. 美军航母舰载机出动回收能力和飞行甲板控制策略探讨［J］. 舰船科学技术， 2020， 42（21）： 174-179. |
| CHEN X F， SHI L P， BI Y Q. Discussion on launch and recovery capability and flight deck control strategy of US aircraft carrier［J］. Ship Science and Technology， 2020， 42（21）： 174-179 （in Chinese）. | |
| [8] | 刘洪铭， 曾鸿雁， 周伟， 等. 基于改进粒子群算法作 业车间调度问题的优化［J］. 山东大学学报（工学版）， 2019， 49（1）： 75-82. |
| LIU H M， ZENG H Y， ZHOU W， et al. Optimization of job shop scheduling problem based on improved particle swarm algorithm［J］. Journal of Shandong University （Engineering Science）， 2019， 49（1）： 75-82 （in Chinese）. | |
| [9] | ELTOUKHY A E E， WANG Z X， CHAN F T S， et al. Joint optimization using a leader-follower Stackelberg game for coordinated configuration of stochastic operational aircraft maintenance routing and maintenance staffing［J］. Computers & Industrial Engineering， 2018， 125： 46-68. |
| [10] | WU J， DAI M Q， WANG J J， et al. Carrier-based aircraft operation support scheduling based on apprenticeship learning algorithm［J］. Chinese Journal of Ship Research， 2022， 17（4）： 145-154. |
| [11] | LI M， YU M. Carrier-based aircraft support operation scheduling based on improved tabu search algorithm［J］. Chinese Journal of Ship Research， 2018， 13（5）： 132-138. |
| [12] | LI M W， QIN N， ZHU T， et al. Carrier aircraft scheduling optimization based on A* algorithm and genetic algorithm［C］∥ 2022 IEEE 11th Data Driven Control and Learning Systems Conference （DDCLS）. Piscataway： IEEE Press， 2022. |
| [13] | FU X Y， CHAN F T S， NIU B， et al. A multi-objective pigeon inspired optimization algorithm for fuzzy production scheduling problem considering mould maintenance［J］. Science China Information Sciences， 2019， 62（7）： 70202. |
| [14] | ZHANG W， DIETTERICH T G. A reinforcement learning approach to job-shop scheduling［C］∥ Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence （IJCAI）. San Francisco： Morgan Kaufmann， 1995. |
| [15] | YAO L W， CHEN Q C， GONG L H， et al. Multi-NPDQ： a multi-agent approach through deep reinforcement learning for operation scheduling［C］∥ International Conference on Intelligent Computing. Singapore： Springer， 2024. |
| [16] | HAO H J， ZHANG X Q， CHI Y， et al. Cooperative carrier aircraft support operation scheduling via multi-agent reinforcement learning［C］∥ 2023 24th IEEE International Conference on Mobile Data Management （MDM）. Piscataway： IEEE Press， 2023. |
| [17] | YAN Q， WANG H F. Double-layer Q-learning-based joint decision-making of dual resource-constrained aircraft assembly scheduling and flexible preventive maintenance［J］. IEEE Transactions on Aerospace and Electronic Systems， 2022， 58（6）： 4938-4952. |
| [18] | ?ALI?IR S， PEHLIVANO?LU M K. Model-free reinforcement learning algorithms： A survey［C］∥ 2019 27th Signal Processing and Communications Applications Conference （SIU）. Piscataway： IEEE Press， 2019. |
| [19] | ARULKUMARAN K， DEISENROTH M P， BRUNDA-GE M， et al. Deep reinforcement learning： A brief survey［J］. IEEE Signal Processing Magazine， 2017， 34（6）： 26-38. |
| [20] | RAVISHANKAR N R， VIJAYAKUMAR M V. Rein-forcement learning algorithms： Survey and classification［J］. Indian Journal of Science and Technology. 2017， 10（1）： 1-8. |
| [21] | WU P， ESCONTRELA A， HAFNER D， et al. Day-dreamer： world models for physical robot learning［C］∥ Conference on Robot Learning. Cambridge： PMLR， 2023. |
| [22] | PAL C V， LEON F. Brief survey of model-based reinforcement learning techniques［C］∥ 2020 24th International Conference on System Theory， Control and Computing （ICSTCC）. Piscataway： IEEE Press， 2020. |
| [23] | PLAAT A， KOSTERS W， PREUSS M. High-accuracy model-based reinforcement learning， a survey［J］. Artificial Intelligence Review， 2023， 56（9）： 9541-9573. |
| [24] | MOERLAND T M， BROEKENS J， PLAAT A， et al. Model-based reinforcement learning： A survey［J］. Foundations and Trends in Machine Learning， 2023， 16（1）： 1-118. |
| [25] | OSINSKI B， FINN C， ERHAN D， et al. Model-based reinforcement learning for atari［C］∥ International Conference on Learning Representations （ICLR）. Amberst： OpenReview， 2020. |
| [26] | 閤泰梓， 唐秋华， 成丽新. 基于 DQN 协同进化算法的柔性作业车间能效调度优化［J］. 计算机集成制造 系统， 2025， 31（2）： 411-422. |
| GE T Z， TANG Q H， CHENG L X. Energy-efficient scheduling optimization for flexible job shop based on DQN co-evolutionary algorithm［J］. Computer Integrated Manufacturing Systems， 2025， 31（2）： 411-422 （in Chinese）. | |
| [27] | MNIH V， KAVUKCUOGLU K， SILVER D， et al. Human-level control through deep reinforcement learning［J］. Nature， 2015， 518（7540）： 529-533. |
| [28] | RYAN J C， BANERJEE A G， CUMMINGS M L， et al. Comparing the performance of expert user heuristics and an integer linear program in aircraft carrier deck operations［J］. IEEE Transactions on Cybernetics， 2014， 44（6）： 761-773. |
| [29] | LI C J， SU X C， ZHANG Y， et al. Integrated scheduling method for fleet wave sorties and maintenance of naval distributed platforms［J］. Advanced Engineering Informatics， 2024， 59： 102340. |
| [30] | 王佳怡， 潘瑞林， 秦飞. 改进遗传算法求解柔性作业车间调度问题［J］. 制造业自动化， 2022， 44（12）： 91-94， 106. |
| WANG J Y， PAN R L， QIN F. Improved genetic algorithm for flexible job shop scheduling problem［J］. Manufacturing Automation， 2022， 44（12）： 91-94， 106 （in Chinese）. | |
| [31] | JIANG T T， SU X C， HAN W. Optimization of support scheduling on deck of carrier aircraft based on improved differential evolution algorithm［C］∥ 2017 3rd IEEE International Conference on Control Science and Systems Engineering （ICCSSE）. Piscataway： IEEE Press， 2017. |
| [32] | CUI R W， HAN W， SU X C， et al. A multi-objective hyper-heuristic framework for integrated optimization of carrier-based aircraft flight deck operations scheduling and resource configuration［J］. Aerospace Science and Technology， 2020， 107： 106346. |
| [33] | TANG C J， ZENG W， ZHAO Z X. Deep reinforcement learning for solving aircraft carrier flight deck tasks scheduling problems［C］∥ Proceedings of the 2024 Guangdong-Hong Kong-Macao Greater Bay Area International Conference on Digital Economy and Artificial Intelligence. New York： ACM， 2024. |
| [34] | LIU R K， PIPLANI R， TORO C. Deep reinforcement learning for dynamic scheduling of a flexible job shop［J］. International Journal of Production Research， 2022， 60（13）： 4049-4069. |
| [35] | LI Y F， WU Q S， XU M L， et al. Real-time scheduling for carrier-borne aircraft support operations： A reinforcement learning approach［J］. Scientia Sinca Informationis， 2021， 51（2）： 247. |
| [36] | HAN W， LI C J， SU X C， et al. Real-time decision support for bolter recovery safety： Long short-term memory network-driven aircraft sequencing［J］. Defence Technology， 2026， 56： 184-205. |
| [37] | LI C J， HAN W， LI H X， et al. Deep reinforcement learning for carrier-based aircraft flight deck operations scheduling problem［J］. Neural Networks， 2026， 200： 108776. |
| [38] | LI C J， HAN W， ZHANG Y， et al. End-to-end scheduling for carrier-based aircraft sortie operations using deep reinforcement learning［J］. Information Processing & Management， 2026， 63（4）： 104590. |
| [39] | LI C J， ZHANG Y， HAN W， et al. Real-time sequencing problem for aircraft recovery on carriers： A safe deep reinforcement learning approach［J］. Chinese Journal of Aeronautics， 2025： 103911. |
| [40] | BALL P， PARKER-HOLDER J， PACCHIANO A， et al. Ready policy one： World building through active learning［C］∥ International Conference on Machine Learning. New York： PMLR， 2020. |
| [41] | HESTER T， QUINLAN M， STONE P. RTMBA： A real-time model-based reinforcement learning architecture for robot control［C］∥ 2012 IEEE International Conference on Robotics and Automation.Piscataway： IEEE Press， 2012. |
| [42] | WU J D， HUANG Z， LV C Y. Uncertainty-aware model-based reinforcement learning： Methodology and application in autonomous driving［J］. IEEE Transactions on Intelligent Vehicles， 2023， 8（1）： 194-203. |
| [43] | IMANBERDIYEV N， FU C H， KAYACAN E， et al. Autonomous navigation of UAV by using real-time model-based reinforcement learning［C］∥ 2016 14th International Conference on Control， Automation， Robotics and Vision （ICARCV）.Piscataway： IEEE Press， 2017. |
| [44] | WILLIAMS G， WAGENER N， GOLDFAIN B， et al. Information theoretic MPC for model-based reinforcement learning［C］∥ 2017 IEEE International Conference on Robotics and Automation （ICRA）. Piscataway： IEEE Press， 2017. |
| [45] | SERBAN I V， SANKAR C， PIEPER M， et al. The bottleneck simulator： A model-based deep reinforcement learning approach［J］. Journal of Artificial Intelligence Research， 2020， 69： 571-612. |
| [46] | 陈旭东， 陈琦琦， 罗祎喆， 等. 异构舰载机舰面保障作业动态并行调度［J］. 航空学报， 2025， 46（13）： 531329. |
| CHEN X D， CHEN Q Q， LUO Y Z， et al. Dynamic parallel scheduling for heterogeneous carrier-based aircraft deck support operations［J］. Acta Aeronautica et Astronautica Sinica， 2025， 46（13）： 531329 （in Chinese）. |
/
| 〈 |
|
〉 |
CJA