中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 多模态人工智能系统全国重点实验室
  5. 多媒体计算与图形学团队
CLIP-VG: Self-Paced Curriculum Adapting of CLIP for Visual Grounding

文献类型：期刊论文

作者Xiao, Linhui1,2,5; Yang, Xiaoshan1,2,5; Peng, Fang1,2,5; Yan, Ming3; Wang, Yaowei4; Xu, Changsheng1,2,5
刊名IEEE TRANSACTIONS ON MULTIMEDIA
出版日期2024
卷号26页码:4334-4347
关键词Grounding Reliability Adaptation models Task analysis Visualization Data models Annotations Visual grounding curriculum learning pseudo-language label and vision-language models
ISSN号1520-9210
DOI10.1109/TMM.2023.3321501
通讯作者Xu, Changsheng(csxu@nlpr.ia.ac.cn)
英文摘要Visual Grounding (VG) is a crucial topic in the field of vision and language, which involves locating a specific region described by expressions within an image. To reduce the reliance on manually labeled data, unsupervised methods have been developed to locate regions using pseudo-labels. However, the performance of existing unsupervised methods is highly dependent on the quality of pseudo-labels and these methods always encounter issues with limited diversity. In order to utilize vision and language pre-trained models to address the grounding problem, and reasonably take advantage of pseudo-labels, we propose CLIP-VG, a novel method that can conduct self-paced curriculum adapting of CLIP with pseudo-language labels. We propose a simple yet efficient end-to-end network architecture to realize the transfer of CLIP to the visual grounding. Based on the CLIP-based architecture, we further propose single-source and multi-source curriculum adapting algorithms, which can progressively find more reliable pseudo-labels to learn an optimal model, thereby achieving a balance between reliability and diversity for the pseudo-language labels. Our method outperforms the current state-of-the-art unsupervised method by a significant margin on RefCOCO/+/g datasets in both single-source and multi-source scenarios, with improvements ranging from 6.78% to 10.67% and 11.39% to 14.87%, respectively. Furthermore, our approach even outperforms existing weakly supervised methods.
资助项目National Natural Science Foundation of China
WOS研究方向Computer Science ; Telecommunications
语种英语
WOS记录号WOS:001181498100046
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
资助机构National Natural Science Foundation of China
源URL[http://ir.ia.ac.cn/handle/173211/56990]  
专题自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队
通讯作者Xu, Changsheng
作者单位1.Peng Cheng Lab PCL, Shenzhen 518066, Peoples R China
2.Univ Chinese Acad Sci UCAS, Sch Artificial Intelligence, Beijing 100049, Peoples R China
3.DAMO Acad, Alibaba Grp, Hangzhou 311121, Peoples R China
4.Peng Cheng Lab, Shenzhen 518066, Peoples R China
5.Chinese Acad Sci CASIA, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714		 Xiao, Linhui,Yang, Xiaoshan,Peng, Fang,et al. CLIP-VG: Self-Paced Curriculum Adapting of CLIP for Visual Grounding[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:4334-4347.
APA Xiao, Linhui,Yang, Xiaoshan,Peng, Fang,Yan, Ming,Wang, Yaowei,&Xu, Changsheng.(2024).CLIP-VG: Self-Paced Curriculum Adapting of CLIP for Visual Grounding.IEEE TRANSACTIONS ON MULTIMEDIA,26,4334-4347.
MLA Xiao, Linhui,et al."CLIP-VG: Self-Paced Curriculum Adapting of CLIP for Visual Grounding".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):4334-4347.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。