|
作者 | Cheng XY(程昕钰)1,2 ; Wei W(魏玮)1 ; Du CD(杜长德)1 ; Qiu S(邱爽)1 ; Tian SL(田三力)2; Ma XJ(马小军)1,2 ; He HG(何晖光)1,2
|
出版日期 | 2022-10
|
会议日期 | 2022-10-10
|
会议地点 | Lisboa, Portugal
|
关键词 | Vigilance Estimation
Multimodal
EEG
EOG
Rapid Serial Visual Presentation
|
英文摘要 |
Recently, brain-computer interface (BCI) technology has made impressive progress and has been developed for many applications. Thereinto, the BCI system based on rapid serial visual presentation (RSVP) is a promising information detection technology. However, the use of RSVP is closely related to the user's performance, which can be influenced by their vigilance levels. Therefore it is crucial to detect vigilance levels in RSVP-based BCI. In this paper, we conducted a long-term RSVP target detection experiment to collect electroencephalography (EEG) and electrooculogram (EOG) data at different vigilance levels. In addition, to estimate vigilance levels in RSVP-based BCI, we propose a multimodal method named VigilanceNet using EEG and EOG. Firstly, we define the multiplicative relationships in conventional EOG features that can better describe the relationships between EOG features, and design an outer product embedding module to extract the multiplicative relationships. Secondly, we propose to decouple the learning of intra- and inter-modality to improve multimodal learning. Specifically, for intra-modality, we introduce an intra-modality representation learning (intra-RL) method to obtain effective representations of each modality by letting each modality independently predict vigilance levels during the multimodal training process. For inter-modality, we employ the cross-modal Transformer based on cross-attention to capture the complementary information between EEG and EOG, which only pays attention to the inter-modality relations. Extensive experiments and ablation studies are conducted on the RSVP and SEED-VIG public datasets. The results demonstrate the effectiveness of the method in terms of regression error and correlation. |
|
语种 | 英语
|
源URL | [http://ir.ia.ac.cn/handle/173211/51834]  |
专题 | 国家专用集成电路设计工程技术研究中心_前瞻芯片研制与测试团队
|
通讯作者 | Ma XJ(马小军); He HG(何晖光) |
作者单位 | 1.中国科学院自动化研究所 2.中国科学院大学
|
推荐引用方式 GB/T 7714 |
Cheng XY,Wei W,Du CD,et al. VigilanceNet: Decouple Intra- and Inter-Modality Learning for Multimodal Vigilance Estimation in RSVP-Based BCI[C]. 见:. Lisboa, Portugal. 2022-10-10.
|