中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 紫东太初大模型研究中心
  5. 图像与视频分析团队
Unbiased Visual Question Answering by Leveraging Instrumental Variable

文献类型：期刊论文

作者Pan, Yonghua1; Liu, Jing2,3; Jin, Lu1; Li, Zechao1
刊名IEEE TRANSACTIONS ON MULTIMEDIA
出版日期2024
卷号26页码:6648-6662
关键词Visualization Correlation Instruments Training Predictive models Color Generators Visual question answering instrumental variable causal inference out of distribution
ISSN号1520-9210
DOI10.1109/TMM.2024.3355640
通讯作者Li, Zechao(zechao.li@njust.edu.cn)
英文摘要Existing unbiased visual question answering (VQA) models reduce the spurious correlation between questions and answers to force the models to focus on visual information. However, the visual information captured by these unbiased models is irrelevant to the correct answer, resulting in leveraging spurious correlation to predict incorrect answers. This makes these unbiased methods fail to obtain critical visual information, thus performing poorly on questions dominated by the visual information. To capture the valuable visual information, this article proposes a novel unbiased VQA model based on causal inference, leveraging Instrumental Variable (IVar) to increase the causal effect between visual features and answers. First, to obtain suitable instrumental variables, the noise generator is proposed according to the constraints of IVar. The generated noise can be regarded as IVar, which is used to pollute the original visual features. Then, this article proposes IVar loss which utilizes the generated IVar to increase the causal effect between visual features and answers. When the visual feature is polluted by IVar, IVar loss guides the model to predict incorrect answers to enhance the correlation between IVar and the answer. Since the correlation between IVar and the answer is proportional to the causal effect between the visual feature and the answer, IVar loss enhances the importance of the visual information, thereby rectifying the model to capture critical visual information. The extensive experimental results on widely-used benchmarks demonstrate the advantages of the proposed method. The proposed method gains the best accuracy on answer type Other of VQA-CP v2. These results demonstrate the superiority of the proposed method in capturing critical visual information since most questions on the answer type Other are dominated by visual information.
资助项目National Key Research and Development Program of China
WOS研究方向Computer Science ; Telecommunications
语种英语
WOS记录号WOS:001200272600018
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
资助机构National Key Research and Development Program of China
源URL[http://ir.ia.ac.cn/handle/173211/58682]  
专题自动化研究所_模式识别国家重点实验室_图像与视频分析团队
通讯作者Li, Zechao
作者单位1.Nanjing Univ Sci & Technol, Sch Comp Sci & Engn, Nanjing 210094, Peoples R China
2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China
推荐引用方式
GB/T 7714		 Pan, Yonghua,Liu, Jing,Jin, Lu,et al. Unbiased Visual Question Answering by Leveraging Instrumental Variable[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:6648-6662.
APA Pan, Yonghua,Liu, Jing,Jin, Lu,&Li, Zechao.(2024).Unbiased Visual Question Answering by Leveraging Instrumental Variable.IEEE TRANSACTIONS ON MULTIMEDIA,26,6648-6662.
MLA Pan, Yonghua,et al."Unbiased Visual Question Answering by Leveraging Instrumental Variable".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):6648-6662.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。