中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Semantic and Relation Modulation for Audio-Visual Event Localization

文献类型：期刊论文

作者Wang, Hao2; Zha, Zheng-Jun2; Li, Liang3; Chen, Xuejin2; Luo, Jiebo1
刊名IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
出版日期2023-06-01
卷号45期号:6页码:7711-7725
ISSN号0162-8828
关键词Visualization Location awareness Correlation Proposals Semantics Task analysis Modulation Audio-visual learning event localization normalization
DOI10.1109/TPAMI.2022.3226328
英文摘要We study the problem of localizing audio-visual events that are both audible and visible in a video. Existing works focus on encoding and aligning audio and visual features at the segment level while neglecting informative correlation between segments of the two modalities and between multi-scale event proposals. We propose a novel Semantic and Relation Modulation Network (SRMN) to learn the above correlation and leverage it to modulate the related auditory, visual, and fused features. In particular, for semantic modulation, we propose intra-modal normalization and cross-modal normalization. The former modulates features of a single modality with the event-relevant semantic guidance of the same modality. The latter modulates features of two modalities by establishing and exploiting the cross-modal relationship. For relation modulation, we propose a multi-scale proposal modulating module and a multi-alignment segment modulating module to introduce multi-scale event proposals and enable dense matching between cross-modal segments, which strengthen correlations between successive segments within one proposal and between all segments. With the features modulated by the correlation information regarding audio-visual events, SRMN performs accurate event localization. Extensive experiments conducted on the public AVE dataset demonstrate that our method outperforms the state-of-the-art methods in both supervised event localization and cross-modality localization tasks.
资助项目National Key R&D Program of China[2020AAA0105702] ; National Natural Science Foundation of China (NSFC)[62225207] ; National Natural Science Foundation of China (NSFC)[U19B2038] ; University Synergy Innovation Program of Anhui Province[GXXT-2019-025]
WOS研究方向Computer Science ; Engineering
语种英语
出版者IEEE COMPUTER SOC
WOS记录号WOS:000982475600073
源URL[http://119.78.100.204/handle/2XEOYT63/21239]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Zha, Zheng-Jun
作者单位1.Univ Rochester, Dept Comp Sci, Rochester, NY 14627 USA
2.Univ Sci & Technol China, Sch Informat Sci & Technol, Hefei 230052, Anhui, Peoples R China
3.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China
推荐引用方式
GB/T 7714		 Wang, Hao,Zha, Zheng-Jun,Li, Liang,et al. Semantic and Relation Modulation for Audio-Visual Event Localization[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(6):7711-7725.
APA Wang, Hao,Zha, Zheng-Jun,Li, Liang,Chen, Xuejin,&Luo, Jiebo.(2023).Semantic and Relation Modulation for Audio-Visual Event Localization.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(6),7711-7725.
MLA Wang, Hao,et al."Semantic and Relation Modulation for Audio-Visual Event Localization".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.6(2023):7711-7725.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。