Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion
文献类型:期刊论文
作者 | Zhang,Yukun1,2; Qiu,Shuang1,2; He,Huiguang1,2 |
刊名 | Journal of Neural Engineering |
出版日期 | 2023-03-13 |
卷号 | 20期号:2页码:026009 |
ISSN号 | 1741-2560 |
关键词 | brain-computer interface motor imagery multimodal EEG-fNIRS center loss |
DOI | 10.1088/1741-2552/acbfdf |
英文摘要 | Abstract Objective. A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. Multimodal MI-BCI that uses multiple neural signals contains rich common and complementary information and is promising for enhancing the decoding accuracy of MI-BCI. However, the heterogeneity of different modalities makes the multimodal decoding task difficult. How to effectively utilize multimodal information remains to be further studied. Approach. In this study, a multimodal MI decoding neural network was proposed. Spatial feature alignment losses were designed to enhance the feature representations extracted from the heterogeneous data and guide the fusion of features from different modalities. An attention-based modality fusion module was built to align and fuse the features in the temporal dimension. To evaluate the proposed decoding method, a five-class MI electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) dataset were constructed. Main results and significance. The comparison experimental results showed that the proposed decoding method achieved higher decoding accuracy than the compared methods on both the self-collected dataset and a public dataset. The ablation results verified the effectiveness of each part of the proposed method. Feature distribution visualization results showed that the proposed losses enhance the feature representation of EEG and fNIRS modalities. The proposed method based on EEG and fNIRS modalities has significant potential for improving decoding performance of MI tasks. |
语种 | 英语 |
出版者 | IOP Publishing |
WOS记录号 | IOP:JNE_20_2_026009 |
源URL | [http://ir.ia.ac.cn/handle/173211/51824] |
专题 | 类脑智能研究中心_神经计算及脑机交互 |
通讯作者 | Qiu,Shuang; He,Huiguang |
作者单位 | 1.Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, People’s Republic of China 2.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, People’s Republic of China |
推荐引用方式 GB/T 7714 | Zhang,Yukun,Qiu,Shuang,He,Huiguang. Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion[J]. Journal of Neural Engineering,2023,20(2):026009. |
APA | Zhang,Yukun,Qiu,Shuang,&He,Huiguang.(2023).Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion.Journal of Neural Engineering,20(2),026009. |
MLA | Zhang,Yukun,et al."Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion".Journal of Neural Engineering 20.2(2023):026009. |
入库方式: OAI收割
来源:自动化研究所
其他版本
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。