Hierarchical Attention Networks for Fact-based Visual Question Answering
文献类型：期刊论文
| 作者 | Yao, Haibo2; Luo, Yongkang1 ; Zhang, Zhi2; Yang, Jianhang2; Cai, Chengtao2
|
| 刊名 | MULTIMEDIA TOOLS AND APPLICATIONS
 |
| 出版日期 | 2023-07-22 |
| 页码 | 18 |
| ISSN号 | 1380-7501 |
| 关键词 | Fact-based Visual Question Answering Hierarchical attention networks Self-attention Multiple attention interaction Positional encoding |
| DOI | 10.1007/s11042-023-16151-w |
| 通讯作者 | Zhang, Zhi(zhangzhi1981@hrbeu.edu.cn) |
| 英文摘要 | Fact-based Visual Question Answering (FVQA) aims to answer questions with images and facts. It requires a fine-grained and simultaneous understanding visual content, textual questions, and factual knowledge. We propose a novel Hierarchical Attention Network (HANet) for FVQA to address the limitations of existing methods. Most existing FVQA methods only consider external facts as a library of answers, which weakens the role of the external facts, and ignore information from images, questions, and external knowledge. Additionally, they only utilize appearance features of images and disregard position information, which results in a model failing to answer many complex questions, due to the absence of important information in images. Our proposed model considers FVQA as a triple modal interaction task and exploits self-attention and multiple attention interaction to make full use of information from all three modalities. In specific, we introduce three attention modules: Self-Attention Layer, Triple-modal Attention Layer, and Bi-Attention Layer to sufficiently extract useful information from images, questions, facts. Furthermore, we also introduce positional encoding into image embedding acquisition to further improve performance of the model. Our proposed method achieves state-of-the-art performance on the FVQA dataset, with top-3 accuracy of 85.98% and top-1 accuracy of 71.68%. |
| WOS关键词 | RECOMMENDATION SYSTEM ; GRAPH |
| 资助项目 | National Key R amp;D Program of China[2019YFE0105400] |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| 出版者 | SPRINGER |
| WOS记录号 | WOS:001034665300003 |
| 资助机构 | National Key R amp;D Program of China |
| 源URL | [http://ir.ia.ac.cn/handle/173211/53897]  |
| 专题 | 多模态人工智能系统全国重点实验室 |
| 通讯作者 | Zhang, Zhi |
| 作者单位 | 1.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China 2.Harbin Engn Univ, Coll Intelligent Syst Sci & Engn, Harbin 150001, Peoples R China |
| 推荐引用方式 GB/T 7714 | Yao, Haibo,Luo, Yongkang,Zhang, Zhi,et al. Hierarchical Attention Networks for Fact-based Visual Question Answering[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2023:18. |
| APA | Yao, Haibo,Luo, Yongkang,Zhang, Zhi,Yang, Jianhang,&Cai, Chengtao.(2023).Hierarchical Attention Networks for Fact-based Visual Question Answering.MULTIMEDIA TOOLS AND APPLICATIONS,18. |
| MLA | Yao, Haibo,et al."Hierarchical Attention Networks for Fact-based Visual Question Answering".MULTIMEDIA TOOLS AND APPLICATIONS (2023):18. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。