基于脑电和眼动信号的人机交互意图识别

王崴; 赵敏睿; 高虹霓; 朱帅; 瞿珏

doi:10.7527/S1000-6893.2020.24290

航空学报 >

2021 , Vol. 42 >Issue 2: 324290 - 324290

DOI: https://doi.org/10.7527/S1000-6893.2020.24290

电子电气工程与控制

基于脑电和眼动信号的人机交互意图识别

王崴 ,
赵敏睿 ,
高虹霓 ,
朱帅 ,
瞿珏

展开

1. 空军工程大学防空反导学院, 西安 710051;
2. 西北工业大学航空学院, 西安 710072

收稿日期: 2020-05-26

修回日期: 2020-06-15

网络出版日期: 2020-07-17

基金资助

国家自然科学基金（51675530）

收起

Human-computer interaction: Intention recognition based on EEG and eye tracking

WANG Wei ,
ZHAO Minrui ,
GAO Hongni ,
ZHU Shuai ,
QU Jue

Expand

1. Air and Missile Defense College, Air Force Engineering University, Xi'an 710051, China;
2. School of Aeronautics, Northwestern Polytechnical University, Xi'an 710072, China

Received date: 2020-05-26

Revised date: 2020-06-15

Online published: 2020-07-17

Supported by

National Natural Science Foundation of China (51675530)

Fold

摘要

意图识别在人机交互（HCI）领域受到广泛关注，传统人机交互意图识别方法单纯依靠脑电（EEG）或眼动数据，不能很好地利用2种方法优点。为此，提出了一种融合脑电和眼动数据的人机交互意图识别方法，通过采集脑电和眼动信号，进行特征提取，输入机器学习模式识别网络进行意图识别，并基于Dempster-Shafer （D-S）证据理论进行决策层融合得出最终识别结果。招募了20名有效受试者进行交互意图识别实验，结果表明，基于脑电和眼动信号的人机交互意图识别方法识别准确率高于单纯依靠脑电和眼动数据的方法，可为下一步飞行器和武器系统人机交互系统自适应设计提供理论依据和技术支持。

关键词： 脑电; 眼动; 意图识别; 人机交互; 数据融合

本文引用格式

王崴 , 赵敏睿 , 高虹霓 , 朱帅 , 瞿珏 . 基于脑电和眼动信号的人机交互意图识别[J]. 航空学报, 2021 , 42(2) : 324290 -324290 . DOI: 10.7527/S1000-6893.2020.24290

Abstract

Intention recognition has received extensive attention in the field of Human-Computer Interaction (HCI). Traditional HCI intention recognition methods rely solely on an electroencephalogram (EEG) or eye movement data without making full use of the advantages of the two methods. This paper proposes an HCI intention recognition method that fuses EEG and eye movement data. It collects EEG and eye movement signals for feature extraction and inputs them into the network of machine learning and pattern recognition for intent recognition. Based on the Dempster-Shafer(D-S) evidence theory, the fusion of the decision layer is performed to obtain the final recognition result. In this study, 20 effective subjects are recruited for interactive intention recognition experiments. Results show that the recognition accuracy of the HCI intention recognition method based on EEG and eye movement signals is higher than that of traditional methods. Therefore, it can provide theoretical and technical support for the adaptive design of the HCI interface in aircraft and weapon equipment systems.

Key words： EEG; eye movement; intention recognition; human-computer interaction; data fusion

参考文献

[1] 罗旭. 控脑技术发展及军事应用预测研究[D]. 重庆:第三军医大学, 2016:53-55. LUO X. Research on the development of brain control technology and military application prediction[D]. Chongqing:Third Military Medical University, 2016:53-55(in Chinese).
[2] GIL M, ALBERT M, FONS J, et al. Designing human-in-the-loop autonomous Cyber-Physical Systems[J]. International Journal of Human-Computer Studies, 2019, 130:21-39.
[3] BENSON P J. Decoding brain-computer interfaces[J]. Science, 2018, 360(6389):615-616.
[4] LEEB R, LEE F Y, KEINRATH C, et al. Brain-computer communication:Motivation, aim, and impact of exploring a virtual apartment[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 15(4):473-482.
[5] GE S, WANG R, LENG Y, et al. A double-partial least-squares model for the detection of steady-state visual evoked potentials[J]. IEEE Journal of Biomedical and Health Informatics, 2017,21(4):897-903.
[6] VAN DOKKUM L, WARD T E, LAFFONT I. Brain computer interfaces for neurorehabilitation-Its current status as a rehabilitation strategy post-stroke[J]. Annals of Physical and Rehabilitation Medicine, 2015, 58(1):3-8.
[7] QIU Z, ALLISON B Z, JIN J, et al. Optimized motor imagery paradigm based on imagining Chinese characters writing movement[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(7):1009-1017.
[8] 张洁, 庞丽萍, 完颜笑如, 等. 基于脑电功率谱密度的作业人员脑力负荷评估方法研究[J]. 航空学报,2020, 41(10):123618. ZHANG J, PANG L P, WANYAN X R, et al. Method for operator mental workload assessment based on power spectral density of EEG[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(10):123618(in Chinese).
[9] 冯传宴, 完颜笑如, 刘双, 等. 负荷条件下注意力分配策略对情境意识的影响[J]. 航空学报, 2020, 41(3):129-138. FENG C Y, WANYAN X R, LIU S, et al. Effects of attention allocation strategies on situational awareness under load conditions[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41(3):129-138(in Chinese).
[10] HAMMER E M, HALDER S, KLEIH S C, et al. Psychological predictors of visual and auditory P300 brain-computer interface performance[J]. Frontiers in Neuroscience, 2018, 12:307-317.
[11] 徐宝国, 何小杭, 魏智唯, 等. 基于运动想象脑电的机器人连续控制系统研究[J]. 仪器仪表学报, 2018, 39(9):10-19. XU B G, HE X H, WEI Z W, et al. Research on robot continuous control system based on motor imagination EEG[J]. Chinese Journal of Scientific Instrument, 2018, 39(9):10-19(in Chinese).
[12] 张力新, 张珊珊, 王坤, 等. 运动相关思维诱发脑电信息解码与应用综述[J].仪器仪表学报, 2019, 40(1):1-11. ZHANG L X, ZHANG S S, WANG K, et al. Review on the decoding and application of motor related thinking induced eeg information[J]. Chinese Journal of Scientific Instrument, 2019,40(1):1-11(in Chinese).
[13] LISI G, NODA T, MORIMOTO J, et al. Decoding the ERD/ERS:Influence of afferent input induced by a leg assistive robot[J]. Frontiers in Systems Neuroscience, 2014, 8:85.
[14] 蒋贵虎, 陈万忠, 马迪, 等.基于ITD和PLV的四类运动想象脑电分类方法研究[J].仪器仪表学报, 2019, 40(5):195-202. JIANG G H, CHEN W Z, MA D, et al. Study on EEG classification methods of four types of motor imagination based on ITD and PLV[J]. Chinese Journal of Scientific Instrument, 2019, 40(5):195-202(in Chinese).
[15] 葛列众.工程心理学[M].上海:华东师范大学出版社, 2017:163-164. GE L Z. Engineering psychology[M]. Shanghai:East China Normal University Press, 2017:163-164(in Chinese).
[16] DENG M M, GU X Z. Information acquisition, emotion experience and behaviour intention during online shopping:An eye-tracking study[J/OL]. Behaviour & Information Technology, (2020-01-13)[2020-04-25]. https://doi.org/10.1080/0144929X.2020.1713890.
[17] 赵其杰, 邵辉, 卢建霞. 基于头眼行为的交互意图检测方法[J]. 仪器仪表学报, 2014, 35(10):2313-2320. ZHAO Q J, SHAO H, LU J X. Interactive intention detection method based on head-eye behavior[J]. Chinese Journal of Scientific Instrument, 2014,35(10):2313-2320(in Chinese).
[18] JIAN M, DONGHEE S. Effects of social popularity and time scarcity on online consumer behaviour regarding smart healthcare products:An eye-tracking approach[J]. Computers in Human Behavior, 2018, 10:78-81.
[19] JARKKO H, CARITA K, YVONNE K. Sixth graders' evaluation strategies when reading internet search results:An eye-tracking study[J]. Behaviour & Information Technology, 2018, 37(8):761-773.
[20] WANG Y, SPARKS B A. An eye-tracking study of tourism photo stimuli:Image characteristics and ethnicity[J]. Journal of Travel Research, 2016, 55(5):588-602.
[21] PARK U, MALLIPEDDI R, LEE M, et al. Human implicit intent discrimination using EEG and eye movement[C]//International Conference on Neural Information Processing, 2014:11-18.
[22] POSTELNICU C C, GIRBACIA F, VOINEA G D, et al. Towards hybrid multimodal brain computer interface for robotic arm command[M]//Augmented cognition. New York:Springer, 2019:217-219.
[23] 谢平, 齐孟松, 张园园, 等. 基于多生理信息及迁移学习的驾驶疲劳评估[J]. 仪器仪表学报, 2018, 39(10):223-231. XIE P, QI M S, ZHANG Y Y, et al. Driving fatigue assessment based on multiple physiological information and transfer learning[J]. Chinese Journal of Scientific Instrument, 2018, 39(10):223-231(in Chinese).
[24] 谢平, 陈迎亚, 郝艳彪, 等. 基于脑肌电融合的混合脑机接口研究[J]. 中国生物医学工程学报, 2016, 35(1):20-30. XIE P, CHEN Y Y, HAO Y B, et al. Multimodal fusion of EEG and EMG signals for a hybrid BCI[J]. Chinese Journal of Biomedical Engineering, 2016, 35(1):20-30(in Chinese).
[25] GIDLÖF K, WALLIN A, DEWHURST R, et al. Using eye tracking to trace a cognitive process:Gaze behaviour during decision making in a natural environment[J]. Journal of Eye Movement Research, 2013, 6(1):613-619.
[26] YOKOYAMA H, EIHATA K, MURAMATSU J, et al. Prediction of driver's workload from slow fluctuations of pupil diameter[C]//2018 IEEE International Conference on Intelligent Transportation Systems (ITSC). Piscataway:IEEE Press, 2018:1775-1780.
[27] 陈丽君, 郑雪. 大学生问题发现过程的眼动研究[J]. 心理学报, 2014, 46(3):367-384. CHEN L J, ZHENG X. An eye movement study on the process of problem discovery in college students[J]. Acta Psychologica Sinica, 2014, 46(3):367-384(in Chinese).
[28] RAYNER K. Eye movements in reading and information processing:20 years of research[J]. Psychological Bulletin, 1998, 124(3):372-422.

Options

文章导航

模态框（Modal）标题

摘要

本文引用格式

Abstract

参考文献