基于改进Q学习的IMA系统重构蓝图生成方法

doi:10.7527/S1000-6893.2021.25792

Abstract

Abstract: Reconfiguration blueprint defines the reconfiguration scheme of system hardware and software resources in the fault status, and is critical to reconfiguration fault tolerance of the integrated modular avionics system. In this paper, we propose an approach for generating reconfiguration blueprints based on improved Q-learning, which considers multiple optimization objectives such as load balance, reconfiguration impact, reconfiguration time, and reconfiguration degradation. The simulated annealing framework is utilized to enhance the convergence performance of the traditional Q-learning strategy. Experimental results demonstrate that compared with the simulated annealing algorithm, the differential evolution algorithm, and the traditional Q-learning algorithm, the algorithm proposed has higher efficiency, and can generate the reconfiguration blueprints of better quality.

Key words: reinforcement learning, Q-learning, simulated annealing algorithm, integrated modular avionics system, multi-objective optimization, reconfiguration

CLC Number:

V242

LUO Qing, ZHANG Tao, SHAN Peng, ZHANG Wentao, LIU Zihao. Generating reconfiguration blueprints for IMA systems based on improved Q-learning[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(8): 525792-525792.

References

[1] PARR G R, EDWARDS R. Integrated modular avionics[J]. Air & Space Europe, 1999, 1(2):72-75.
[2] SUO D J, AN J X, ZHU J H. A new approach to improve safety of reconfiguration in Integrated Modular Avionics[C]//2011 IEEE/AIAA 30th Digital Avionics Systems Conference. Piscataway:IEEE Press, 2011:1C4-1.
[3] FU J P, WANG S H, LIU B. An original approach to constructing test model for IMA blueprints[C]//2017 Second International Conference on Reliability Systems Engineering (ICRSE). Piscataway:IEEE Press, 2017:1-6.
[4] JOLLIFFE G, NICHOLSON M. Exploring the possibilities towards a preliminary safety case for IMA blueprints[C]//Constituents of Modern System-Safety Thinking, 2005.
[5] ZHOU T R, XIONQ H, ZHANG Z. Hierarchical resource allocation for integrated modular avionics systems[J]. Journal of Systems Engineering and Electronics, 2011, 22(5):780-787.
[6] MONTANA D, HUSSAIN T, VIDAVER G. A genetic-algorithm-based reconfigurable scheduler[M]//Evolutionary Scheduling, 2007:577-611.
[7] GIRAULT A, KALLA H, SIGHIREANU M, et al. An algorithm for automatically obtaining distributed and fault-tolerant static schedules[C]//2003 International Conference on Dependable Systems and Networks. Piscataway:IEEE Press, 2003:159-168.
[8] OMIECINSKI T A. Reconfigurable integrated modular avionics[D]. Bristol:University of Bristol, 1999.
[9] XU J H, PU H J, SUN Z K. Study of fault tolerance design for Integrated Modular Avionics system[C]//Proceedings of Korean Aerospace Society Academic Conference, 2011:1710-1714.
[10] SUTTON R S, BARTO A G. Reinforcement learning:An introduction[J]. IEEE Transactions on Neural Networks, 1998, 9(5):1054.
[11] WATKINS C J C H, DAYAN P. Q-learning[J]. Machine Learning, 1992, 8(3-4):279-292.
[12] WANG R, PURSHOUSE R C, FLEMING P J. Preference-inspired coevolutionary algorithms for many-objective optimization[J]. IEEE Transactions on Evolutionary Computation, 2013, 17(4):474-494.
[13] 刘若辰, 李建霞, 刘静, 等. 动态多目标优化研究综述[J]. 计算机学报, 2020, 43(7):1246-1278. LIU R C, LI J X, LIU J, et al. A survey on dynamic multi-objective optimization[J]. Chinese Journal of Computers, 2020, 43(7):1246-1278(in Chinese).
[14] CUI Y Q, SHI J Y, WANG Z L. Backward reconfiguration management for modular avionic reconfigurable systems[J]. IEEE Systems Journal, 2018, 12(1):137-148.
[15] WANG R P, LU W T, ZENG C H, et al. Reliability modeling and verification method for dynamic reconfiguration system[C]//2018 Prognostics and System Health Management Conference (PHM-Chongqing). Piscataway:IEEE Press, 2018:941-947.
[16] WEI X M, DONG Y W, XIAO M R. Safety-based software reconfiguration method for integrated modular avionics systems in AADL model[C]//2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). Piscataway:IEEE Press, 2018:450-455.
[17] POURMOHSENI B, WILDERMANN S, GLAß M, et al. Hard real-time application mapping reconfiguration for NoC-based many-core systems[J]. Real-Time Systems, 2019, 55(2):433-469.
[18] ZHANG Q, WANG S H, LIU B. Approach for integrated modular avionics reconfiguration modelling and reliability analysis based on AADL[J]. IET Software, 2016, 10(1):18-25.
[19] DA FONTOURA A A, DO NASCIMENTO F A M, NADJM-TEHRANI S, et al. Timing assurance of avionic reconfiguration schemes using formal analysis[J]. IEEE Transactions on Aerospace and Electronic Systems, 2020, 56(1):95-106.
[20] GUO R, ZHONG D M, SUN R, et al. Optimized design of resource sharing and isolation for integrated modular avionics[C]//2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS). Piscataway:IEEE Press, 2019:444-447.
[21] CLEMENTE J A, RAMO E P, RESANO J, et al. Configuration mapping algorithms to reduce energy and time reconfiguration overheads in reconfigurable systems[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2014, 22(6):1248-1261.
[22] WEGENER I. Simulated annealing beats metropolis in combinatorial optimization[C]//Automata, Languages and Programming, 2005.
[23] SINGH H K, RAY T, SMITH W. C-PSA:Constrained Pareto simulated annealing for constrained multi-objective optimization[J]. Information Sciences, 2010, 180(13):2499-2513.
[24] ZHANG Z S, XING L N, CHEN Y N, et al. Evolutionary algorithms for many-objective ground station scheduling problem[C]//Bio-Inspired Computing-Theories and Applications, 2016.
[25] ZHANG T, CHEN J Y, LV D, et al. Automatic generation of reconfiguration blueprints for IMA systems using reinforcement learning[J]. IEEE Embedded Systems Letters, 2021, 99:1-4.
[26] CHUAI G, ZHAO D, SUN L. Novel adaptive simulated annealing algorithm for constrained multi-objective optimization[J]. China Communications, 2012, 9(9):68-78(in Chinese).
[27] CHENG Y, WANG H X, WENG Z Y, et al. Optimization of flow shop scheduling control strategy based on improved differential evolution algorithm[C]//2018 33rd Youth Academic Annual Conference of Chinese Association of Automation (YAC). Piscataway:IEEE Press, 2018:43-46.

Generating reconfiguration blueprints for IMA systems based on improved Q-learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Chao LIANG, Li YANG, Chengsheng PAN, Yaowen QI. Multi-QoS constraints routing algorithm based on satellite network dynamic resource graph [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2023, 44(1): 326422-326422.
[2]	ZOU Ziyuan, CHEN Qifeng. Decision tree-based target assignment for confrontation of multiple space vehicles [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 726910-726910.
[3]	HUI Junpeng, WANG Ren, YU Qidong. Generating new quality flight corridor for reentry aircraft based on reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 325960-325960.
[4]	SUN Gang, PENG Shuang, CHEN Hao, WU Jiangjiang, LI Jun. Multi-objective optimization method oriented to integrated scenario of TT & C resources and data transmission resources [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 326114-326114.
[5]	ZHENG Shuai, WANG Zihan, ZHAO Haoran, YANG Pengtao, HONG Jun. Layout optimization method of aircraft fuel gauging sensor based on differential evolution algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(8): 125809-125809.
[6]	FU Xiaowei, WANG Hui, XU Zhe. Cooperative pursuit strategy for multi-UAVs based on DE-MADDPG algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(5): 325311-325311.
[7]	JING Tao, TIAN Xitian. Multi-objective optimization method for aircraft tolerance allocation based on Monte Carlo-adaptive differential evolution algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(3): 425278-425278.
[8]	ZHOU Bin, GUO Yan, LI Ning, ZHONG Xijian. Path planning of UAV using guided enhancement Q-learning algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(9): 325109-325109.
[9]	SUN Zhixiao, YANG Shengqi, PIAO Haiyin, BAI Chengchao, GE Jun. A survey of air combat artificial intelligence [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(8): 525799-525799.
[10]	ZHANG Junfeng, YOU Lubao, YANG Chunwei, HU Rong. Arrival sequencing and scheduling based on multi-objective Imperialist competitive algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(6): 324439-324439.
[11]	YANG Jianan, HOU Xiaolei, HU Yu Hen, LIU Yong, PAN Quan, FENG Qian. Heuristic enhanced reinforcement learning method for large-scale multi-debris active removal mission planning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524354-524354.
[12]	XIANG Xiaojia, YAN Chao, WANG Chang, YIN Dong. Coordination control method for fixed-wing UAV formation through deep reinforcement learning [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524009-524009.
[13]	REN Feng, GAO Chuanqiang, TANG Hui. Machine learning for flow control: Applications and development trends [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524686-524686.
[14]	SUN Gang, CHEN Hao, PENG Shuang, DU Chun, LI Jun. Multi-objective optimization algorithm for satellite range scheduling based on preference MOEA [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524475-524475.
[15]	LI Runze, ZHANG Yufei, CHEN Haixin. Reinforcement learning method for supercritical airfoil aerodynamic design [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 523810-523810.