中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 智能感知与计算研究中心
Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain

文献类型：期刊论文

作者Shi, Qianqian1,3; Fan, Junsong2; Wang, Zuoren1,3; Zhang, Zhaoxiang1,2
刊名PATTERN RECOGNITION
出版日期2022-10-01
卷号130页码:14
关键词Multisensory integration Top-down attention Multimodal transformer Fine-grained bird recognition Emotion recognition
ISSN号0031-3203
DOI10.1016/j.patcog.2022.108837
通讯作者Wang, Zuoren(zuorenwang@ion.ac.cn) ; Zhang, Zhaoxiang(zhaoxiang.zhang@ia.ac.cn)
英文摘要Multisensory integration has attracted intense studies for decades. How to combine visual and auditory information to optimize perception and decision-making is a key question in neuroscience as well as machine learning. Inspired by the mechanisms of multisensory integration in the brain, we propose a multimodal channel-wise attention transformer (MCAT) that performs reliability-weighted integration and revises the weights allocation according to a top-down attention-like mechanism. We apply MCAT on EFLSTM neural networks for a fine-grained video bird recognition task, and on MulT neural networks for an emotion recognition task. The performance of both models is improved remarkably. Ablation study shows that the attention mechanism is indispensable for effective multisensory integration. Moreover, we found that cross-modal integration models are in accordance with the law of inverse effectiveness of multisensory integration in the brain, which reveals that our model may have mechanisms similar to those in the brain. Taken together, the results demonstrate that the brain-inspired MCAT block is effective for improving multisensory integration, providing useful clues for designing new algorithms and understanding multisensory integration in the brain. (C) 2022 Elsevier Ltd. All rights reserved.
WOS关键词NEURONAL OSCILLATIONS ; MODEL
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:000833526200004
出版者ELSEVIER SCI LTD
源URL[http://ir.ia.ac.cn/handle/173211/49824]  
专题自动化研究所_智能感知与计算研究中心
通讯作者Wang, Zuoren; Zhang, Zhaoxiang
作者单位1.Univ Chinese Acad Sci, Sch Future Technol, Beijing 100049, Peoples R China
2.Chinese Acad Sci, Ctr Res Intelligent Percept & Comp CRIPAC, Inst Automat, Natl Lab Pattern Recognit NLPR, Beijing 100190, Peoples R China
3.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Inst Neurosci, State Key Lab Neurosci, Shanghai 200031, Peoples R China
推荐引用方式
GB/T 7714		 Shi, Qianqian,Fan, Junsong,Wang, Zuoren,et al. Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain[J]. PATTERN RECOGNITION,2022,130:14.
APA Shi, Qianqian,Fan, Junsong,Wang, Zuoren,&Zhang, Zhaoxiang.(2022).Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain.PATTERN RECOGNITION,130,14.
MLA Shi, Qianqian,et al."Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain".PATTERN RECOGNITION 130(2022):14.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。