Event Graph Guided Compositional Spatial--Temporal Reasoning for Video Question Answering
文献类型：期刊论文
| 作者 | Bai, Ziyi1,2; Wang, Ruiping1,2; Gao, Difei1,2; Chen, Xilin1,2 |
| 刊名 | IEEE TRANSACTIONS ON IMAGE PROCESSING
 |
| 出版日期 | 2024 |
| 卷号 | 33页码:1109-1121 |
| 关键词 | Visualization Cognition Transformers Semantics Feature extraction Context modeling Task analysis VideoQA video representation transformer spatial-temporal reasoning compositional reasoning |
| ISSN号 | 1057-7149 |
| DOI | 10.1109/TIP.2024.3358726 |
| 英文摘要 | Video question answering (VideoQA) is challenging since it requires the model to extract and combine multi-level visual concepts from local objects to global actions from complex events for compositional reasoning. Existing works represent the video with fixed-duration clip features that make the model struggle in capturing the crucial concepts in multiple granularities. To overcome this shortcoming, we propose to represent the video with an Event Graph in a hierarchical structure whose nodes correspond to visual concepts of different levels (object, relation, scene and action) and edges indicate their spatial-temporal relationships. We further propose a Hierarchical S patial- Temporal Transformer (HSTT) which takes nodes from the graph as visual input to realize compositional reasoning guided by the event graph. To fully exploit the spatial-temporal context delivered from the graph structure, on the one hand, we encode the nodes in the order of their semantic hierarchy (depth) and occurrence time (breadth) with our improved graph search algorithm; On the other hand, we introduce edge-guided attention to combine the spatial-temporal context among nodes according to their edge connections. HSTT then performs QA by cross-modal interactions guaranteed by the hierarchical correspondence between the multi-level event graph and the cross-level question. Experiments on the recent challenging AGQA and STAR datasets show that the proposed method clearly outperforms the existing VideoQA models by a large margin, including those pre-trained with large-scale external data. |
| 资助项目 | National Key Research and Development Program of China |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:001174109400006 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/38786]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Wang, Ruiping |
| 作者单位 | 1.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 2.Chinese Acad Sci, Key Lab Intelligent Informat Proc, Inst Comp Technol, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Bai, Ziyi,Wang, Ruiping,Gao, Difei,et al. Event Graph Guided Compositional Spatial--Temporal Reasoning for Video Question Answering[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2024,33:1109-1121. |
| APA | Bai, Ziyi,Wang, Ruiping,Gao, Difei,&Chen, Xilin.(2024).Event Graph Guided Compositional Spatial--Temporal Reasoning for Video Question Answering.IEEE TRANSACTIONS ON IMAGE PROCESSING,33,1109-1121. |
| MLA | Bai, Ziyi,et al."Event Graph Guided Compositional Spatial--Temporal Reasoning for Video Question Answering".IEEE TRANSACTIONS ON IMAGE PROCESSING 33(2024):1109-1121. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。