GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion
文献类型：期刊论文
| 作者 | Huang, Wuliang; Chen, Yiqiang; Jiang, Xinlong; Zhang, Teng; Chen, Qian |
| 刊名 | ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS
 |
| 出版日期 | 2024-02-01 |
| 卷号 | 20期号:2页码:23 |
| 关键词 | Multimodal physiological signal graph neural network emotion state recognition ubiquitous computing |
| ISSN号 | 1551-6857 |
| DOI | 10.1145/3617503 |
| 英文摘要 | Physiological signal based ubiquitous computing has garnered significant attention. However, the heterogeneity among multimodal physiological signals poses a critical challenge to practical applications. To traverse this heterogeneity gap, recent studies have focused on establishing inter-modality correlations. Early works only consider coarse-level correlations between the embeddings of each modality. More recent graph-based approaches incorporate prior knowledge-based correlations, although they may not be entirely accurate. In this article, we propose the Graph Joint Fusion (GJFusion) network, which leverages channel-level inter-modality correlations based on a graph joint to mitigate the heterogeneous gap. Our proposed GJFusion first represents each modality as a graph, with each vertex corresponding to a signal channel, and the edges denoting their functional connectivity. We then join each modality by constructing inter-modality correlations for each salient channel using a sampling-based matching method. Discarded channels are transformed into a virtual vertex through a lightweight pooling operation. Subsequently, the fusion network integrates intra- and inter-modality features, enabling multimodal physiological signal fusion. To validate the effectiveness of our method, we select emotional state recognition as the downstream task and conduct comprehensive experiments on two benchmark datasets. The results demonstrate that our proposed GJFusion network surpasses the latest state-of-the-art methods, achieving relative accuracy improvements of 1.22% and 0.81% on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, visualization experiments of the salient brain regions reveal the presence of interpretable knowledge within the proposed GJFusion model. |
| 资助项目 | National Key Research and Development Plan of China[2021YFC2501202] ; Beijing Municipal Science & Technology Commission[Z221100002722009] ; Hunan Provincial Natural Science Foundation of China[2023JJ70034] ; Youth Innovation Promotion Association CAS |
| WOS研究方向 | Computer Science |
| 语种 | 英语 |
| WOS记录号 | WOS:001092595800030 |
| 出版者 | ASSOC COMPUTING MACHINERY |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/38101]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Chen, Yiqiang; Jiang, Xinlong |
| 作者单位 | Chinese Acad Sci, Inst Comp Technol, 6 Kexueyuan South Rd Zhongguancun, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Huang, Wuliang,Chen, Yiqiang,Jiang, Xinlong,et al. GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion[J]. ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS,2024,20(2):23. |
| APA | Huang, Wuliang,Chen, Yiqiang,Jiang, Xinlong,Zhang, Teng,&Chen, Qian.(2024).GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion.ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS,20(2),23. |
| MLA | Huang, Wuliang,et al."GJFusion: A Channel-Level Correlation Construction Method for Multimodal Physiological Signal Fusion".ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 20.2(2024):23. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。