基于改进Q学习的IMA系统重构蓝图生成方法

doi:10.7527/S1000-6893.2021.25792

论文

本期目录 | 过刊浏览 | 高级检索

前一篇 | 后一篇

基于改进Q学习的IMA系统重构蓝图生成方法

罗庆^1,2, 张涛³, 单鹏⁴, 张文涛³, 刘子豪³

1. 航空工业沈阳飞机设计研究所, 沈阳 110035;
2. 南京航空航天大学航天学院, 南京 210016;
3. 西北工业大学软件学院, 西安 710072;
4. 航空工业西安航空计算技术研究所, 西安 710065

收稿日期:2021-04-15 修回日期:2021-05-08 发布日期:2021-05-31
通讯作者: 张涛 E-mail:tao_zhang@nwpu.edu.cn
基金资助:
航空科学基金（2015ZD53055，20185853038，201907053004）

Generating reconfiguration blueprints for IMA systems based on improved Q-learning

LUO Qing^1,2, ZHANG Tao³, SHAN Peng⁴, ZHANG Wentao³, LIU Zihao³

1. AVIC Shenyang Aircraft Design and Research Institute, Shenyang 110035, China;
2. School of Aviation, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China;
3. School of Software, Northwestern Polytechnical University, Xi'an 710072, China;
4. AVIC Xi'an Institute of Aeronautical Computing Technology, Xi'an 710065, China

Received:2021-04-15 Revised:2021-05-08 Published:2021-05-31
Supported by:
Aeronautical Science Foundation of China (2015ZD53055, 20185853038, 201907053004)

摘要/Abstract

摘要： 重构蓝图定义了故障状态下系统软硬件资源的重新配置方案，是实现综合模块化航空电子系统重构容错的关键。提出了一种基于改进Q学习的重构蓝图生成方法，综合考虑负载均衡、重构影响、重构时间、重构降级等多优化目标，并应用模拟退火框架改进探索策略，提高了传统Q学习算法的收敛性能。实验结果表明，与模拟退火算法、差分进化算法、传统Q学习算法相比，本文提出的改进Q学习算法效率更高，所生成重构蓝图质量更高。

关键词: 强化学习, Q学习, 模拟退火算法, 综合模块化航空电子系统, 多目标优化, 重构

Abstract: Reconfiguration blueprint defines the reconfiguration scheme of system hardware and software resources in the fault status, and is critical to reconfiguration fault tolerance of the integrated modular avionics system. In this paper, we propose an approach for generating reconfiguration blueprints based on improved Q-learning, which considers multiple optimization objectives such as load balance, reconfiguration impact, reconfiguration time, and reconfiguration degradation. The simulated annealing framework is utilized to enhance the convergence performance of the traditional Q-learning strategy. Experimental results demonstrate that compared with the simulated annealing algorithm, the differential evolution algorithm, and the traditional Q-learning algorithm, the algorithm proposed has higher efficiency, and can generate the reconfiguration blueprints of better quality.

Key words: reinforcement learning, Q-learning, simulated annealing algorithm, integrated modular avionics system, multi-objective optimization, reconfiguration

中图分类号:

V242

罗庆, 张涛, 单鹏, 张文涛, 刘子豪. 基于改进Q学习的IMA系统重构蓝图生成方法[J]. 航空学报, 2021, 42(8): 525792-525792.

LUO Qing, ZHANG Tao, SHAN Peng, ZHANG Wentao, LIU Zihao. Generating reconfiguration blueprints for IMA systems based on improved Q-learning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(8): 525792-525792.

参考文献

[1] PARR G R, EDWARDS R. Integrated modular avionics[J]. Air & Space Europe, 1999, 1(2):72-75.
[2] SUO D J, AN J X, ZHU J H. A new approach to improve safety of reconfiguration in Integrated Modular Avionics[C]//2011 IEEE/AIAA 30th Digital Avionics Systems Conference. Piscataway:IEEE Press, 2011:1C4-1.
[3] FU J P, WANG S H, LIU B. An original approach to constructing test model for IMA blueprints[C]//2017 Second International Conference on Reliability Systems Engineering (ICRSE). Piscataway:IEEE Press, 2017:1-6.
[4] JOLLIFFE G, NICHOLSON M. Exploring the possibilities towards a preliminary safety case for IMA blueprints[C]//Constituents of Modern System-Safety Thinking, 2005.
[5] ZHOU T R, XIONQ H, ZHANG Z. Hierarchical resource allocation for integrated modular avionics systems[J]. Journal of Systems Engineering and Electronics, 2011, 22(5):780-787.
[6] MONTANA D, HUSSAIN T, VIDAVER G. A genetic-algorithm-based reconfigurable scheduler[M]//Evolutionary Scheduling, 2007:577-611.
[7] GIRAULT A, KALLA H, SIGHIREANU M, et al. An algorithm for automatically obtaining distributed and fault-tolerant static schedules[C]//2003 International Conference on Dependable Systems and Networks. Piscataway:IEEE Press, 2003:159-168.
[8] OMIECINSKI T A. Reconfigurable integrated modular avionics[D]. Bristol:University of Bristol, 1999.
[9] XU J H, PU H J, SUN Z K. Study of fault tolerance design for Integrated Modular Avionics system[C]//Proceedings of Korean Aerospace Society Academic Conference, 2011:1710-1714.
[10] SUTTON R S, BARTO A G. Reinforcement learning:An introduction[J]. IEEE Transactions on Neural Networks, 1998, 9(5):1054.
[11] WATKINS C J C H, DAYAN P. Q-learning[J]. Machine Learning, 1992, 8(3-4):279-292.
[12] WANG R, PURSHOUSE R C, FLEMING P J. Preference-inspired coevolutionary algorithms for many-objective optimization[J]. IEEE Transactions on Evolutionary Computation, 2013, 17(4):474-494.
[13] 刘若辰, 李建霞, 刘静, 等. 动态多目标优化研究综述[J]. 计算机学报, 2020, 43(7):1246-1278. LIU R C, LI J X, LIU J, et al. A survey on dynamic multi-objective optimization[J]. Chinese Journal of Computers, 2020, 43(7):1246-1278(in Chinese).
[14] CUI Y Q, SHI J Y, WANG Z L. Backward reconfiguration management for modular avionic reconfigurable systems[J]. IEEE Systems Journal, 2018, 12(1):137-148.
[15] WANG R P, LU W T, ZENG C H, et al. Reliability modeling and verification method for dynamic reconfiguration system[C]//2018 Prognostics and System Health Management Conference (PHM-Chongqing). Piscataway:IEEE Press, 2018:941-947.
[16] WEI X M, DONG Y W, XIAO M R. Safety-based software reconfiguration method for integrated modular avionics systems in AADL model[C]//2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). Piscataway:IEEE Press, 2018:450-455.
[17] POURMOHSENI B, WILDERMANN S, GLAß M, et al. Hard real-time application mapping reconfiguration for NoC-based many-core systems[J]. Real-Time Systems, 2019, 55(2):433-469.
[18] ZHANG Q, WANG S H, LIU B. Approach for integrated modular avionics reconfiguration modelling and reliability analysis based on AADL[J]. IET Software, 2016, 10(1):18-25.
[19] DA FONTOURA A A, DO NASCIMENTO F A M, NADJM-TEHRANI S, et al. Timing assurance of avionic reconfiguration schemes using formal analysis[J]. IEEE Transactions on Aerospace and Electronic Systems, 2020, 56(1):95-106.
[20] GUO R, ZHONG D M, SUN R, et al. Optimized design of resource sharing and isolation for integrated modular avionics[C]//2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS). Piscataway:IEEE Press, 2019:444-447.
[21] CLEMENTE J A, RAMO E P, RESANO J, et al. Configuration mapping algorithms to reduce energy and time reconfiguration overheads in reconfigurable systems[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2014, 22(6):1248-1261.
[22] WEGENER I. Simulated annealing beats metropolis in combinatorial optimization[C]//Automata, Languages and Programming, 2005.
[23] SINGH H K, RAY T, SMITH W. C-PSA:Constrained Pareto simulated annealing for constrained multi-objective optimization[J]. Information Sciences, 2010, 180(13):2499-2513.
[24] ZHANG Z S, XING L N, CHEN Y N, et al. Evolutionary algorithms for many-objective ground station scheduling problem[C]//Bio-Inspired Computing-Theories and Applications, 2016.
[25] ZHANG T, CHEN J Y, LV D, et al. Automatic generation of reconfiguration blueprints for IMA systems using reinforcement learning[J]. IEEE Embedded Systems Letters, 2021, 99:1-4.
[26] CHUAI G, ZHAO D, SUN L. Novel adaptive simulated annealing algorithm for constrained multi-objective optimization[J]. China Communications, 2012, 9(9):68-78(in Chinese).
[27] CHENG Y, WANG H X, WENG Z Y, et al. Optimization of flow shop scheduling control strategy based on improved differential evolution algorithm[C]//2018 33rd Youth Academic Annual Conference of Chinese Association of Automation (YAC). Piscataway:IEEE Press, 2018:43-46.

E-mail：hkxb@buaa.edu.cn

关于我们

期刊社服务

专业学科

封面文章

友情链接

主管单位：中国科学技术协会主办单位：中国航空学会北京航空航天大学

基于改进Q学习的IMA系统重构蓝图生成方法

Generating reconfiguration blueprints for IMA systems based on improved Q-learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张鸿林, 罗建军, 马卫华. 基于机器学习的航天器规避目标威胁博弈决策[J]. 航空学报, 2024, 45(8): 329136-329136.
[2]	蔡云鹏, 周大鹏, 丁江川. 具有防撞安全约束的无人机集群智能协同控制[J]. 航空学报, 2024, 45(5): 529683-529683.
[3]	单圣哲, 张伟伟. 基于自博弈深度强化学习的空战智能决策方法[J]. 航空学报, 2024, 45(4): 328723-328723.
[4]	柳汀, 林麒, 刘震, 王晓光, 吴惠松, 许勇刚. 电磁散射测试的可重构绳系并联支撑机构[J]. 航空学报, 2024, 45(2): 328658-328658.
[5]	张桂玮, 刘召庆, 朱镭, 张衡, 田玮, 李伟光, 杨智春. 地面颤振模拟试验技术研究进展[J]. 航空学报, 2024, 45(10): 29229-029229.
[6]	杨加秀, 李新凯, 张宏立, 王昊. 切换拓扑下异构集群的强化学习时变编队控制[J]. 航空学报, 2024, 45(10): 329166-329166.
[7]	肖冰, 张海朝. 航天器姿态稳定强化学习鲁棒最优控制方法[J]. 航空学报, 2024, 45(1): 628890-628890.
[8]	魏雁昕, 刘君. 一种基于面元梯度重建的格心型有限体积空间离散方法[J]. 航空学报, 2024, 45(1): 128541-128541.
[9]	倪炜霖, 王永海, 徐聪, 赤丰华, 梁海朝. 基于强化学习的高超飞行器协同博弈制导方法[J]. 航空学报, 2023, 44(S2): 729400-729400.
[10]	杨倩, 郑皓冉, 程显达, 董威. 基于引气控制的热气防冰优化设计方法[J]. 航空学报, 2023, 44(S2): 729285-729285.
[11]	王雪鉴, 文永明, 石晓荣, 张宁宁, 刘洁玺. 多智能体多耦合任务混合式智能决策架构设计[J]. 航空学报, 2023, 44(S2): 729770-729770.
[12]	邓有朋, 范佳宣, 郑岩, 王振亚, 吕勇梁, 李雨霄. 不完全信息下多智能体对手建模[J]. 航空学报, 2023, 44(S2): 729782-729782.
[13]	范之琳, 杨洪勇, 韩艺琳. 基于强化学习的多智能体系统目标围捕控制[J]. 航空学报, 2023, 44(S1): 727487-727487.
[14]	王宏越, 王兵, 方国东, 孟松鹤. 2.5D机织浅交弯联复合材料数字单元建模分析[J]. 航空学报, 2023, 44(9): 227478-227478.
[15]	符小卫, 徐哲, 朱金冬, 王楠. 基于PER-MATD3的多无人机攻防对抗机动决策[J]. 航空学报, 2023, 44(7): 327083-327083.