基于脑电和眼动信号的人机交互意图识别

doi:10.7527/S1000-6893.2020.24290

Abstract

Abstract: Intention recognition has received extensive attention in the field of Human-Computer Interaction (HCI). Traditional HCI intention recognition methods rely solely on an electroencephalogram (EEG) or eye movement data without making full use of the advantages of the two methods. This paper proposes an HCI intention recognition method that fuses EEG and eye movement data. It collects EEG and eye movement signals for feature extraction and inputs them into the network of machine learning and pattern recognition for intent recognition. Based on the Dempster-Shafer(D-S) evidence theory, the fusion of the decision layer is performed to obtain the final recognition result. In this study, 20 effective subjects are recruited for interactive intention recognition experiments. Results show that the recognition accuracy of the HCI intention recognition method based on EEG and eye movement signals is higher than that of traditional methods. Therefore, it can provide theoretical and technical support for the adaptive design of the HCI interface in aircraft and weapon equipment systems.

Key words: EEG, eye movement, intention recognition, human-computer interaction, data fusion

CLC Number:

V7
R857.14

WANG Wei, ZHAO Minrui, GAO Hongni, ZHU Shuai, QU Jue. Human-computer interaction: Intention recognition based on EEG and eye tracking[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(2): 324290-324290.

References

[1] 罗旭. 控脑技术发展及军事应用预测研究[D]. 重庆:第三军医大学, 2016:53-55. LUO X. Research on the development of brain control technology and military application prediction[D]. Chongqing:Third Military Medical University, 2016:53-55(in Chinese).
[2] GIL M, ALBERT M, FONS J, et al. Designing human-in-the-loop autonomous Cyber-Physical Systems[J]. International Journal of Human-Computer Studies, 2019, 130:21-39.
[3] BENSON P J. Decoding brain-computer interfaces[J]. Science, 2018, 360(6389):615-616.
[4] LEEB R, LEE F Y, KEINRATH C, et al. Brain-computer communication:Motivation, aim, and impact of exploring a virtual apartment[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 15(4):473-482.
[5] GE S, WANG R, LENG Y, et al. A double-partial least-squares model for the detection of steady-state visual evoked potentials[J]. IEEE Journal of Biomedical and Health Informatics, 2017,21(4):897-903.
[6] VAN DOKKUM L, WARD T E, LAFFONT I. Brain computer interfaces for neurorehabilitation-Its current status as a rehabilitation strategy post-stroke[J]. Annals of Physical and Rehabilitation Medicine, 2015, 58(1):3-8.
[7] QIU Z, ALLISON B Z, JIN J, et al. Optimized motor imagery paradigm based on imagining Chinese characters writing movement[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(7):1009-1017.
[8] 张洁, 庞丽萍, 完颜笑如, 等. 基于脑电功率谱密度的作业人员脑力负荷评估方法研究[J]. 航空学报,2020, 41(10):123618. ZHANG J, PANG L P, WANYAN X R, et al. Method for operator mental workload assessment based on power spectral density of EEG[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(10):123618(in Chinese).
[9] 冯传宴, 完颜笑如, 刘双, 等. 负荷条件下注意力分配策略对情境意识的影响[J]. 航空学报, 2020, 41(3):129-138. FENG C Y, WANYAN X R, LIU S, et al. Effects of attention allocation strategies on situational awareness under load conditions[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(3):129-138(in Chinese).
[10] HAMMER E M, HALDER S, KLEIH S C, et al. Psychological predictors of visual and auditory P300 brain-computer interface performance[J]. Frontiers in Neuroscience, 2018, 12:307-317.
[11] 徐宝国, 何小杭, 魏智唯, 等. 基于运动想象脑电的机器人连续控制系统研究[J]. 仪器仪表学报, 2018, 39(9):10-19. XU B G, HE X H, WEI Z W, et al. Research on robot continuous control system based on motor imagination EEG[J]. Chinese Journal of Scientific Instrument, 2018, 39(9):10-19(in Chinese).
[12] 张力新, 张珊珊, 王坤, 等. 运动相关思维诱发脑电信息解码与应用综述[J].仪器仪表学报, 2019, 40(1):1-11. ZHANG L X, ZHANG S S, WANG K, et al. Review on the decoding and application of motor related thinking induced eeg information[J]. Chinese Journal of Scientific Instrument, 2019,40(1):1-11(in Chinese).
[13] LISI G, NODA T, MORIMOTO J, et al. Decoding the ERD/ERS:Influence of afferent input induced by a leg assistive robot[J]. Frontiers in Systems Neuroscience, 2014, 8:85.
[14] 蒋贵虎, 陈万忠, 马迪, 等.基于ITD和PLV的四类运动想象脑电分类方法研究[J].仪器仪表学报, 2019, 40(5):195-202. JIANG G H, CHEN W Z, MA D, et al. Study on EEG classification methods of four types of motor imagination based on ITD and PLV[J]. Chinese Journal of Scientific Instrument, 2019, 40(5):195-202(in Chinese).
[15] 葛列众.工程心理学[M].上海:华东师范大学出版社, 2017:163-164. GE L Z. Engineering psychology[M]. Shanghai:East China Normal University Press, 2017:163-164(in Chinese).
[16] DENG M M, GU X Z. Information acquisition, emotion experience and behaviour intention during online shopping:An eye-tracking study[J/OL]. Behaviour & Information Technology, (2020-01-13)[2020-04-25]. https://doi.org/10.1080/0144929X.2020.1713890.
[17] 赵其杰, 邵辉, 卢建霞. 基于头眼行为的交互意图检测方法[J]. 仪器仪表学报, 2014, 35(10):2313-2320. ZHAO Q J, SHAO H, LU J X. Interactive intention detection method based on head-eye behavior[J]. Chinese Journal of Scientific Instrument, 2014,35(10):2313-2320(in Chinese).
[18] JIAN M, DONGHEE S. Effects of social popularity and time scarcity on online consumer behaviour regarding smart healthcare products:An eye-tracking approach[J]. Computers in Human Behavior, 2018, 10:78-81.
[19] JARKKO H, CARITA K, YVONNE K. Sixth graders' evaluation strategies when reading internet search results:An eye-tracking study[J]. Behaviour & Information Technology, 2018, 37(8):761-773.
[20] WANG Y, SPARKS B A. An eye-tracking study of tourism photo stimuli:Image characteristics and ethnicity[J]. Journal of Travel Research, 2016, 55(5):588-602.
[21] PARK U, MALLIPEDDI R, LEE M, et al. Human implicit intent discrimination using EEG and eye movement[C]//International Conference on Neural Information Processing, 2014:11-18.
[22] POSTELNICU C C, GIRBACIA F, VOINEA G D, et al. Towards hybrid multimodal brain computer interface for robotic arm command[M]//Augmented cognition. New York:Springer, 2019:217-219.
[23] 谢平, 齐孟松, 张园园, 等. 基于多生理信息及迁移学习的驾驶疲劳评估[J]. 仪器仪表学报, 2018, 39(10):223-231. XIE P, QI M S, ZHANG Y Y, et al. Driving fatigue assessment based on multiple physiological information and transfer learning[J]. Chinese Journal of Scientific Instrument, 2018, 39(10):223-231(in Chinese).
[24] 谢平, 陈迎亚, 郝艳彪, 等. 基于脑肌电融合的混合脑机接口研究[J]. 中国生物医学工程学报, 2016, 35(1):20-30. XIE P, CHEN Y Y, HAO Y B, et al. Multimodal fusion of EEG and EMG signals for a hybrid BCI[J]. Chinese Journal of Biomedical Engineering, 2016, 35(1):20-30(in Chinese).
[25] GIDLÖF K, WALLIN A, DEWHURST R, et al. Using eye tracking to trace a cognitive process:Gaze behaviour during decision making in a natural environment[J]. Journal of Eye Movement Research, 2013, 6(1):613-619.
[26] YOKOYAMA H, EIHATA K, MURAMATSU J, et al. Prediction of driver's workload from slow fluctuations of pupil diameter[C]//2018 IEEE International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018:1775-1780.
[27] 陈丽君, 郑雪. 大学生问题发现过程的眼动研究[J]. 心理学报, 2014, 46(3):367-384. CHEN L J, ZHENG X. An eye movement study on the process of problem discovery in college students[J]. Acta Psychologica Sinica, 2014, 46(3):367-384(in Chinese).
[28] RAYNER K. Eye movements in reading and information processing:20 years of research[J]. Psychological Bulletin, 1998, 124(3):372-422.

Human-computer interaction: Intention recognition based on EEG and eye tracking

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	SU Lingfei, HUA Yongzhao, DONG Xiwang, REN Zhang. Human-UAV swarm multi-modal intelligent interaction methods [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(S1): 727001-727001.
[2]	CAO Ming, WANG Peng, ZUO Hongfu, ZENG Haijun, SUN Jianzhong, YANG Weidong, WEI Fang, CHEN Xuefeng. Current status, challenges and opportunities of civil aero-engine diagnostics & health management Ⅱ: Comprehensive off-board diagnosis, life management and intelligent condition based MRO [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(9): 625574-625574.
[3]	LEI Shiying, SUN Jianzhong, LIU He. Cumulative damage index model and service reliability evaluation of turbine blade [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2022, 43(3): 225064-225064.
[4]	ZHANG Weiwei, KOU Jiaqing, LIU Yilang. Prospect of artificial intelligence empowered fluid mechanics [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(4): 524689-524689.
[5]	HE Qiang, TAN Deqiang, CHENG Lin. Eye movement and fatigue detection in visual inspection of civil aircraft composite materials [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(5): 223532-223532.
[6]	FENG Chuanyan, WANYAN Xiaoru, LIU Shuang, CHEN Hao, ZHUANG Damin, WANG Xin. Influence of different attention allocation strategies under workloads on situation awareness [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(3): 123307-123307.
[7]	CHANG Qi, YANG Weixi, ZHAO Heng, MENG Yao, LIU Jun, GAO Heming. A multi-sensor based crack propagation monitoring research [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2020, 41(2): 223336-223336.
[8]	YE Lijun, LIU Fucheng, YIN Haining, XU Ying, BAOYIN Hexi. Online estimation of low frequency error for star tracker based on vertical filter [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2019, 40(10): 323163-323163.
[9]	BU Jian, LIU Yinxin, WANG Yanjun. Relationship between air traffic controllers' eye movement and fatigue [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2017, 38(S1): 721525-721525.
[10]	TIAN Wei, WANG Yue, SHAN Xiuming, YANG Jian. A Track-to-track Association Algorithm Based on Maximizing the Consistent Association Number [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2014, 35(11): 3115-3122.
[11]	FENG Xiaoxue, LIANG Yan, JIAO Lianmeng, ZHOU Lin. Mode Identification and Positioning Accuracy Improvement Using Forward-based Receivers for OTHR [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(11): 2590-2598.
[12]	ZHOU Lai, ZHENG Danli, GU Hongbin, WANG Debao. Research of Visual Interaction for Virtual Reality Flight Training [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2013, 34(10): 2391-2401.
[13]	KONG Ruonan, CHENG Yongmei, LIANG Yan, WANG Zengfu, SUI Zhijia. Mode Identification and Location Improvement Based on Data Association of OTHR and Azimuth-only Sensors [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2012, (6): 1061-1069.
[14]	Huang Hanyan;Wang Zhengming. Acquisition of Damage Response Function Based on Optimal Design and Dimensional Method [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2009, 30(12): 2354-2362.
[15]	Wu Zemin;Ren Shujie;Liu Xi. Topology Sequence Based Track Correlation Algorithm [J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2009, 30(10): 1937-1942.