基于脑电和眼动信号的人机交互意图识别

doi:10.7527/S1000-6893.2020.24290

电子电气工程与控制

本期目录 | 过刊浏览 | 高级检索

前一篇 | 后一篇

基于脑电和眼动信号的人机交互意图识别

王崴¹, 赵敏睿¹, 高虹霓¹, 朱帅¹, 瞿珏^1,2

1. 空军工程大学防空反导学院, 西安 710051;
2. 西北工业大学航空学院, 西安 710072

收稿日期:2020-05-26 修回日期:2020-06-15 发布日期:2020-07-17
通讯作者: 瞿珏 E-mail:qujue402@sina.com
基金资助:
国家自然科学基金（51675530）

Human-computer interaction: Intention recognition based on EEG and eye tracking

WANG Wei¹, ZHAO Minrui¹, GAO Hongni¹, ZHU Shuai¹, QU Jue^1,2

1. Air and Missile Defense College, Air Force Engineering University, Xi'an 710051, China;
2. School of Aeronautics, Northwestern Polytechnical University, Xi'an 710072, China

Received:2020-05-26 Revised:2020-06-15 Published:2020-07-17
Supported by:
National Natural Science Foundation of China (51675530)

摘要/Abstract

摘要： 意图识别在人机交互（HCI）领域受到广泛关注，传统人机交互意图识别方法单纯依靠脑电（EEG）或眼动数据，不能很好地利用2种方法优点。为此，提出了一种融合脑电和眼动数据的人机交互意图识别方法，通过采集脑电和眼动信号，进行特征提取，输入机器学习模式识别网络进行意图识别，并基于Dempster-Shafer （D-S）证据理论进行决策层融合得出最终识别结果。招募了20名有效受试者进行交互意图识别实验，结果表明，基于脑电和眼动信号的人机交互意图识别方法识别准确率高于单纯依靠脑电和眼动数据的方法，可为下一步飞行器和武器系统人机交互系统自适应设计提供理论依据和技术支持。

关键词: 脑电, 眼动, 意图识别, 人机交互, 数据融合

Abstract: Intention recognition has received extensive attention in the field of Human-Computer Interaction (HCI). Traditional HCI intention recognition methods rely solely on an electroencephalogram (EEG) or eye movement data without making full use of the advantages of the two methods. This paper proposes an HCI intention recognition method that fuses EEG and eye movement data. It collects EEG and eye movement signals for feature extraction and inputs them into the network of machine learning and pattern recognition for intent recognition. Based on the Dempster-Shafer(D-S) evidence theory, the fusion of the decision layer is performed to obtain the final recognition result. In this study, 20 effective subjects are recruited for interactive intention recognition experiments. Results show that the recognition accuracy of the HCI intention recognition method based on EEG and eye movement signals is higher than that of traditional methods. Therefore, it can provide theoretical and technical support for the adaptive design of the HCI interface in aircraft and weapon equipment systems.

Key words: EEG, eye movement, intention recognition, human-computer interaction, data fusion

中图分类号:

V7
R857.14

王崴, 赵敏睿, 高虹霓, 朱帅, 瞿珏. 基于脑电和眼动信号的人机交互意图识别[J]. 航空学报, 2021, 42(2): 324290-324290.

WANG Wei, ZHAO Minrui, GAO Hongni, ZHU Shuai, QU Jue. Human-computer interaction: Intention recognition based on EEG and eye tracking[J]. ACTA AERONAUTICAET ASTRONAUTICA SINICA, 2021, 42(2): 324290-324290.

参考文献

[1] 罗旭. 控脑技术发展及军事应用预测研究[D]. 重庆:第三军医大学, 2016:53-55. LUO X. Research on the development of brain control technology and military application prediction[D]. Chongqing:Third Military Medical University, 2016:53-55(in Chinese).
[2] GIL M, ALBERT M, FONS J, et al. Designing human-in-the-loop autonomous Cyber-Physical Systems[J]. International Journal of Human-Computer Studies, 2019, 130:21-39.
[3] BENSON P J. Decoding brain-computer interfaces[J]. Science, 2018, 360(6389):615-616.
[4] LEEB R, LEE F Y, KEINRATH C, et al. Brain-computer communication:Motivation, aim, and impact of exploring a virtual apartment[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 15(4):473-482.
[5] GE S, WANG R, LENG Y, et al. A double-partial least-squares model for the detection of steady-state visual evoked potentials[J]. IEEE Journal of Biomedical and Health Informatics, 2017,21(4):897-903.
[6] VAN DOKKUM L, WARD T E, LAFFONT I. Brain computer interfaces for neurorehabilitation-Its current status as a rehabilitation strategy post-stroke[J]. Annals of Physical and Rehabilitation Medicine, 2015, 58(1):3-8.
[7] QIU Z, ALLISON B Z, JIN J, et al. Optimized motor imagery paradigm based on imagining Chinese characters writing movement[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(7):1009-1017.
[8] 张洁, 庞丽萍, 完颜笑如, 等. 基于脑电功率谱密度的作业人员脑力负荷评估方法研究[J]. 航空学报,2020, 41(10):123618. ZHANG J, PANG L P, WANYAN X R, et al. Method for operator mental workload assessment based on power spectral density of EEG[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(10):123618(in Chinese).
[9] 冯传宴, 完颜笑如, 刘双, 等. 负荷条件下注意力分配策略对情境意识的影响[J]. 航空学报, 2020, 41(3):129-138. FENG C Y, WANYAN X R, LIU S, et al. Effects of attention allocation strategies on situational awareness under load conditions[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(3):129-138(in Chinese).
[10] HAMMER E M, HALDER S, KLEIH S C, et al. Psychological predictors of visual and auditory P300 brain-computer interface performance[J]. Frontiers in Neuroscience, 2018, 12:307-317.
[11] 徐宝国, 何小杭, 魏智唯, 等. 基于运动想象脑电的机器人连续控制系统研究[J]. 仪器仪表学报, 2018, 39(9):10-19. XU B G, HE X H, WEI Z W, et al. Research on robot continuous control system based on motor imagination EEG[J]. Chinese Journal of Scientific Instrument, 2018, 39(9):10-19(in Chinese).
[12] 张力新, 张珊珊, 王坤, 等. 运动相关思维诱发脑电信息解码与应用综述[J].仪器仪表学报, 2019, 40(1):1-11. ZHANG L X, ZHANG S S, WANG K, et al. Review on the decoding and application of motor related thinking induced eeg information[J]. Chinese Journal of Scientific Instrument, 2019,40(1):1-11(in Chinese).
[13] LISI G, NODA T, MORIMOTO J, et al. Decoding the ERD/ERS:Influence of afferent input induced by a leg assistive robot[J]. Frontiers in Systems Neuroscience, 2014, 8:85.
[14] 蒋贵虎, 陈万忠, 马迪, 等.基于ITD和PLV的四类运动想象脑电分类方法研究[J].仪器仪表学报, 2019, 40(5):195-202. JIANG G H, CHEN W Z, MA D, et al. Study on EEG classification methods of four types of motor imagination based on ITD and PLV[J]. Chinese Journal of Scientific Instrument, 2019, 40(5):195-202(in Chinese).
[15] 葛列众.工程心理学[M].上海:华东师范大学出版社, 2017:163-164. GE L Z. Engineering psychology[M]. Shanghai:East China Normal University Press, 2017:163-164(in Chinese).
[16] DENG M M, GU X Z. Information acquisition, emotion experience and behaviour intention during online shopping:An eye-tracking study[J/OL]. Behaviour & Information Technology, (2020-01-13)[2020-04-25]. https://doi.org/10.1080/0144929X.2020.1713890.
[17] 赵其杰, 邵辉, 卢建霞. 基于头眼行为的交互意图检测方法[J]. 仪器仪表学报, 2014, 35(10):2313-2320. ZHAO Q J, SHAO H, LU J X. Interactive intention detection method based on head-eye behavior[J]. Chinese Journal of Scientific Instrument, 2014,35(10):2313-2320(in Chinese).
[18] JIAN M, DONGHEE S. Effects of social popularity and time scarcity on online consumer behaviour regarding smart healthcare products:An eye-tracking approach[J]. Computers in Human Behavior, 2018, 10:78-81.
[19] JARKKO H, CARITA K, YVONNE K. Sixth graders' evaluation strategies when reading internet search results:An eye-tracking study[J]. Behaviour & Information Technology, 2018, 37(8):761-773.
[20] WANG Y, SPARKS B A. An eye-tracking study of tourism photo stimuli:Image characteristics and ethnicity[J]. Journal of Travel Research, 2016, 55(5):588-602.
[21] PARK U, MALLIPEDDI R, LEE M, et al. Human implicit intent discrimination using EEG and eye movement[C]//International Conference on Neural Information Processing, 2014:11-18.
[22] POSTELNICU C C, GIRBACIA F, VOINEA G D, et al. Towards hybrid multimodal brain computer interface for robotic arm command[M]//Augmented cognition. New York:Springer, 2019:217-219.
[23] 谢平, 齐孟松, 张园园, 等. 基于多生理信息及迁移学习的驾驶疲劳评估[J]. 仪器仪表学报, 2018, 39(10):223-231. XIE P, QI M S, ZHANG Y Y, et al. Driving fatigue assessment based on multiple physiological information and transfer learning[J]. Chinese Journal of Scientific Instrument, 2018, 39(10):223-231(in Chinese).
[24] 谢平, 陈迎亚, 郝艳彪, 等. 基于脑肌电融合的混合脑机接口研究[J]. 中国生物医学工程学报, 2016, 35(1):20-30. XIE P, CHEN Y Y, HAO Y B, et al. Multimodal fusion of EEG and EMG signals for a hybrid BCI[J]. Chinese Journal of Biomedical Engineering, 2016, 35(1):20-30(in Chinese).
[25] GIDLÖF K, WALLIN A, DEWHURST R, et al. Using eye tracking to trace a cognitive process:Gaze behaviour during decision making in a natural environment[J]. Journal of Eye Movement Research, 2013, 6(1):613-619.
[26] YOKOYAMA H, EIHATA K, MURAMATSU J, et al. Prediction of driver's workload from slow fluctuations of pupil diameter[C]//2018 IEEE International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018:1775-1780.
[27] 陈丽君, 郑雪. 大学生问题发现过程的眼动研究[J]. 心理学报, 2014, 46(3):367-384. CHEN L J, ZHENG X. An eye movement study on the process of problem discovery in college students[J]. Acta Psychologica Sinica, 2014, 46(3):367-384(in Chinese).
[28] RAYNER K. Eye movements in reading and information processing:20 years of research[J]. Psychological Bulletin, 1998, 124(3):372-422.

E-mail：hkxb@buaa.edu.cn

关于我们

期刊社服务

专业学科

封面文章

友情链接

主管单位：中国科学技术协会主办单位：中国航空学会北京航空航天大学

基于脑电和眼动信号的人机交互意图识别

Human-computer interaction: Intention recognition based on EEG and eye tracking

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	杨华, 陈树生, 高正红, 姜权峰, 张伟. 基于贝叶斯框架的旋翼气动力数据融合[J]. 航空学报, 2024, 45(8): 128960-128960.
[2]	田阔, 孙志勇, 李增聪. 面向结构静力试验监测的高精度数字孪生方法[J]. 航空学报, 2024, 45(7): 429134-429134.
[3]	宋海浪, 张建东, 史国庆, 杨啟明, 张耀中. 航空火控系统试飞综合评估技术与方法探讨[J]. 航空学报, 2024, 45(5): 529687-529687.
[4]	刘鹏宇, 朱雪耀. 基于深度学习的融合空域空管指令语义解析技术[J]. 航空学报, 2023, 44(S1): 727592-727592.
[5]	张怀捷, 马静雅, 刘浩源, 郭品, 邓慧超, 徐坤, 丁希仑. 视觉与惯性融合的多旋翼飞行机器人室内定位技术[J]. 航空学报, 2023, 44(5): 426964-426964.
[6]	丁鹏, 宋亚飞. 代价敏感的空中目标意图识别方法[J]. 航空学报, 2023, 44(24): 328551-328551.
[7]	苏翎菲, 化永朝, 董希旺, 任章. 人与无人机集群多模态智能交互方法[J]. 航空学报, 2022, 43(S1): 727001-727001.
[8]	雷世英, 孙见忠, 刘赫. 涡轮叶片累积损伤指数模型及服役可靠性评估[J]. 航空学报, 2022, 43(3): 225064-225064.
[9]	张伟伟, 寇家庆, 刘溢浪. 智能赋能流体力学展望[J]. 航空学报, 2021, 42(4): 524689-524689.
[10]	贺强, 谭德强, 程林. 民机复合材料目视检查中的眼动与疲劳检测[J]. 航空学报, 2020, 41(5): 223532-223532.
[11]	冯传宴, 完颜笑如, 刘双, 陈浩, 庄达民, 王鑫. 负荷条件下注意力分配策略对情境意识的影响[J]. 航空学报, 2020, 41(3): 123307-123307.
[12]	常琦, 杨维希, 赵恒, 孟瑶, 刘君, 高鹤明. 基于多传感器的裂纹扩展监测研究[J]. 航空学报, 2020, 41(2): 223336-223336.
[13]	叶立军, 刘付成, 尹海宁, 徐樱, 宝音贺西. 基于纵向滤波的星敏感器低频误差在线估计[J]. 航空学报, 2019, 40(10): 323163-323163.
[14]	卜建, 刘银鑫, 王艳军. 空中交通管制员的眼动行为与疲劳关系[J]. 航空学报, 2017, 38(S1): 721525-721525.
[15]	田威, 王钺, 山秀明, 杨健. 基于一致关联数最大化的航迹关联算法[J]. 航空学报, 2014, 35(11): 3115-3122.