中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
Patching the visual ability of large multimodal models by collaborating with small models

文献类型：期刊论文

作者Liang, Hao1,3; Zhang, Xiaolong1,3; Kan, Meina1,3; Shan, Shiguang1,2,3; Chen, Xilin1,3
刊名FRONTIERS OF COMPUTER SCIENCE
出版日期2026-02-12
卷号20期号:9页码:17
关键词model collaboration patching visual ability large multimodal models
ISSN号2095-2228
DOI10.1007/s11704-025-41126-5
英文摘要Large multimodal models (LMMs) have demonstrated significant success across various tasks but fall short on some basic visual functions, such as inaccurate object counting and imprecise localization. These limitations restrict the application of LMMs in broad scenarios. To enhance the capabilities of LMMs, we propose a novel method to patch their visual perceptual abilities by collaborating with small task-specific models. Our method begins with utilizing an LMM to decompose the user query into a series of visual functions. For each function, the appropriate model, either the LMM itself or a small task-specific model, is invoked. To determine whether to patch the LMM with a small task-specific model, we design a novel question-answering-based reinforcement learning strategy to optimize the decision process. Finally, the LMM generates the answer utilizing the visual perceptual results. The proposed method is evaluated on two standard visual question-answering datasets and two specialized datasets. The experimental results demonstrate that our method effectively enhances the visual abilities of LMMs.
资助项目National Natural Science Foundation of China[62495082] ; National Natural Science Foundation of China[U2336213]
WOS研究方向Computer Science
语种英语
WOS记录号WOS:001690415600001
出版者HIGHER EDUCATION PRESS
源URL[http://119.78.100.204/handle/2XEOYT63/42795]  
专题中国科学院计算技术研究所
通讯作者Kan, Meina
作者单位1.Chinese Acad Sci, Inst Comp Technol, State Key Lab AI Safety, Beijing 100190, Peoples R China
2.Peng Cheng Natl Lab, Shenzhen 518055, Peoples R China
3.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式
GB/T 7714		 Liang, Hao,Zhang, Xiaolong,Kan, Meina,et al. Patching the visual ability of large multimodal models by collaborating with small models[J]. FRONTIERS OF COMPUTER SCIENCE,2026,20(9):17.
APA Liang, Hao,Zhang, Xiaolong,Kan, Meina,Shan, Shiguang,&Chen, Xilin.(2026).Patching the visual ability of large multimodal models by collaborating with small models.FRONTIERS OF COMPUTER SCIENCE,20(9),17.
MLA Liang, Hao,et al."Patching the visual ability of large multimodal models by collaborating with small models".FRONTIERS OF COMPUTER SCIENCE 20.9(2026):17.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。