Dual-Stream Recurrent Neural Network for Video Captioning
文献类型：期刊论文
| 作者 | Xu, Ning5; Liu, An-An5; Wong, Yongkang1; Zhang, Yongdong2,3; Nie, Weizhi5; Su, Yuting5; Kankanhalli, Mohan4 |
| 刊名 | IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
 |
| 出版日期 | 2019-08-01 |
| 卷号 | 29期号:8页码:2482-2493 |
| ISSN号 | 1051-8215 |
| 关键词 | Video captioning hidden state fusion dual stream recurrent neural network attention module |
| DOI | 10.1109/TCSVT.2018.2867286 |
| 英文摘要 | Recent progress in using recurrent neural networks (RNNs) for video description has attracted an increasing interest, due to its capability to encode a sequence of frames for caption generation. While existing methods have studied various features (e.g., CNN, 3D CNN, and semantic attributes) for visual encoding, the representation and fusion of heterogeneous information from multi-modal spaces have not fully explored. Consider that different modalities are often asynchronous, frame-level multi-modal fusion (e.g., concatenation and linear fusion) will negatively influence each modality. In this paper, we propose a dual-stream RNN (DS-RNN) framework to jointly discover and integrate the hidden states of both visual and semantic streams for video caption generation. First, an encoding RNN is used for each stream to flexibly exploit the hidden states of respective modality. Specifically, we proposed an attentive multi-grained encoder module to enhance the local feature learning with global semantics feature. Then, a dual-stream decoder is deployed to integrate the asynchronous yet complementary sequential hidden states from both streams for caption generation. Extensive experiments on three benchmark datasets, namely, MSVD, MSR-VTT, and MPII-MD, show that DS-RNN achieves competitive performance against the state-of-the-art. Additional ablation studies were conducted on various variants of the proposed DS-RNN. |
| 资助项目 | National Natural Science Foundation of China[61772359] ; National Natural Science Foundation of China[61472275] ; National Natural Science Foundation of China[61525206] ; National Natural Science Foundation of China[61502337] ; National Key Research and Development Program of China[2017YFC0820600] ; National Defense Science and Technology Fund for Distinguished Young Scholars[2017-JCJQ-ZQ-022] ; National Research Foundation, Prime Minister's Office, Singapore, under its International Research Centre in Singapore Funding Initiative |
| WOS研究方向 | Engineering |
| 语种 | 英语 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| WOS记录号 | WOS:000480310500022 |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/4434]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Liu, An-An |
| 作者单位 | 1.Natl Univ Singapore, Smart Syst Inst, Singapore 119077, Singapore 2.Univ Sci & Technol China, Sch Informat Sci & Technol, Hefei 230026, Anhui, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 4.Natl Univ Singapore, Sch Comp, Singapore 119077, Singapore 5.Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China |
| 推荐引用方式 GB/T 7714 | Xu, Ning,Liu, An-An,Wong, Yongkang,et al. Dual-Stream Recurrent Neural Network for Video Captioning[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(8):2482-2493. |
| APA | Xu, Ning.,Liu, An-An.,Wong, Yongkang.,Zhang, Yongdong.,Nie, Weizhi.,...&Kankanhalli, Mohan.(2019).Dual-Stream Recurrent Neural Network for Video Captioning.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(8),2482-2493. |
| MLA | Xu, Ning,et al."Dual-Stream Recurrent Neural Network for Video Captioning".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.8(2019):2482-2493. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。