中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Bi-Modality Individual-Aware Prompt Tuning for Visual-Language Model

文献类型：期刊论文

作者Yao, Hantao3; Zhang, Rui4; Lyu, Huaihai1,2; Zhang, Yongdong3; Xu, Changsheng1,2
刊名IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
出版日期2025-08-01
卷号47期号:8页码:6352-6368
关键词Tuning Visualization Training Adaptation models Hands Feature extraction Data models Artificial intelligence Transformers Data mining Prompt tuning individual-aware prompt tuning bi-modality prompt tuning visual-language model
ISSN号0162-8828
DOI10.1109/TPAMI.2025.3557780
英文摘要Prompt tuning is a valuable technique for adapting visual language models (VLMs) to different downstream tasks, such as domain generalization and learning from a few examples. Previous methods have utilized Context Optimization approaches to deduce domain-shared or cross-modality prompt tokens, which enhance generalization and discriminative ability in textual or visual contexts. However, these prompt tokens, inferred from training data, cannot adapt perfectly to the distribution of the test dataset. This work introduces a novel approach called Bi-modality Individual-aware Prompt Tuning (BIP) by explicitly incorporating the individual's essential prior knowledge into the learnable prompt to enhance their discriminability and generalization. The critical insight of BIP involves applying the Textual Knowledge Embedding (TKE) and Visual Knowledge Embedding (VKE) models to project the class-aware textual essential knowledge and the instance-aware essential knowledge into the class-aware prompt and instance-aware prompt, referred to as Textual-level Class-aware Prompt tuning (TCP) and Visual-level Instance-aware Prompt tuning (VIP). On the one hand, TCP integrates the generated class-aware prompts into the Text Encoder to produce a dynamic class-aware classifier to improve generalization on unseen domains. On the other hand, VIP uses the instance-aware prompt to generate the dynamic visual embedding of each instance, thereby enhancing the discriminative capability of visual embedding. Comprehensive evaluations demonstrate that BIP can be used as a plug-and-play module easily integrated with existing methods and achieves superior performance on 15 benchmarks across four tasks.
资助项目National Science and Technology[2021ZD0112202] ; National Natural Science Foundation of China[62376268] ; National Natural Science Foundation of China[U23A20387] ; National Natural Science Foundation of China[U21B2044] ; National Natural Science Foundation of China[62121002]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:001522958700031
出版者IEEE COMPUTER SOC
源URL[http://119.78.100.204/handle/2XEOYT63/41766]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Yao, Hantao
作者单位1.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
3.Univ Sci & Technol China, Sch Informat Sci & Technol, Hefei 230026, Peoples R China
4.Chinese Acad Sci, Inst Comp Technol, State Key Lab Proc, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714		 Yao, Hantao,Zhang, Rui,Lyu, Huaihai,et al. Bi-Modality Individual-Aware Prompt Tuning for Visual-Language Model[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2025,47(8):6352-6368.
APA Yao, Hantao,Zhang, Rui,Lyu, Huaihai,Zhang, Yongdong,&Xu, Changsheng.(2025).Bi-Modality Individual-Aware Prompt Tuning for Visual-Language Model.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,47(8),6352-6368.
MLA Yao, Hantao,et al."Bi-Modality Individual-Aware Prompt Tuning for Visual-Language Model".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 47.8(2025):6352-6368.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。