中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
MemBridge: Video-Language Pre-Training With Memory-Augmented Inter-Modality Bridge

文献类型：期刊论文

作者Yang, Jiahao1,3; Li, Xiangyang1,3; Zheng, Mao2; Wang, Zihan1,3; Zhu, Yongqing1,3; Guo, Xiaoqian1,3; Yuan, Yuchen2; Chai, Zifeng2; Jiang, Shuqiang1,3
刊名IEEE TRANSACTIONS ON IMAGE PROCESSING
出版日期2023
卷号32页码:4073-4087
关键词Video-language pre-training inter-modality bridge memory module
ISSN号1057-7149
DOI10.1109/TIP.2023.3283916
英文摘要Video-language pre-training has attracted considerable attention recently for its promising performance on various downstream tasks. Most existing methods utilize the modality-specific or modality-joint representation architectures for the cross-modality pre-training. Different from previous methods, this paper presents a novel architecture named Memory-augmented Inter-Modality Bridge (MemBridge), which uses the learnable intermediate modality representations as the bridge for the interaction between videos and language. Specifically, in the transformer-based cross-modality encoder, we introduce the learnable bridge tokens as the interaction approach, which means the video and language tokens can only perceive information from bridge tokens and themselves. Moreover, a memory bank is proposed to store abundant modality interaction information for adaptively generating bridge tokens according to different cases, enhancing the capacity and robustness of the inter-modality bridge. Through pre-training, MemBridge explicitly models the representations for more sufficient inter-modality interaction. Comprehensive experiments show that our approach achieves competitive performance with previous methods on various downstream tasks including video-text retrieval, video captioning, and video question answering on multiple datasets, demonstrating the effectiveness of the proposed method. The code has been available at https://github.com/jahhaoyang/MemBridge.
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:001033515600013
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
源URL[http://119.78.100.204/handle/2XEOYT63/21341]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Jiang, Shuqiang
作者单位1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
2.Tencent Co Ltd, Dept Machine Learning Platform, Beijing 100193, Peoples R China
3.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式
GB/T 7714		 Yang, Jiahao,Li, Xiangyang,Zheng, Mao,et al. MemBridge: Video-Language Pre-Training With Memory-Augmented Inter-Modality Bridge[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2023,32:4073-4087.
APA Yang, Jiahao.,Li, Xiangyang.,Zheng, Mao.,Wang, Zihan.,Zhu, Yongqing.,...&Jiang, Shuqiang.(2023).MemBridge: Video-Language Pre-Training With Memory-Augmented Inter-Modality Bridge.IEEE TRANSACTIONS ON IMAGE PROCESSING,32,4073-4087.
MLA Yang, Jiahao,et al."MemBridge: Video-Language Pre-Training With Memory-Augmented Inter-Modality Bridge".IEEE TRANSACTIONS ON IMAGE PROCESSING 32(2023):4073-4087.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。