中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Improving multimodal named entity recognition via text-image relevance prediction with large language models

文献类型：期刊论文

作者Zeng, Qingyang1; Yuan, Minghui1; Su, Yueyang2; Mi, Jia1; Che, Qianzi3; Wan, Jing1
刊名NEUROCOMPUTING
出版日期2025-10-28
卷号651页码:10
关键词Multimodal named entity recognition Multimodal learning Large language model Contrastive learning Social media
ISSN号0925-2312
DOI10.1016/j.neucom.2025.130982
英文摘要Multimodal Named Entity Recognition (MNER) is a critical task in information extraction, which aims to identify named entities in text-image pairs and classify them into specific types such as person, organization and location. While existing studies have achieved moderate success by fusing visual and textual features through cross-modal attention mechanisms, two major challenges remain: (1) image-text mismatch, where the two modalities are not always semantically aligned in real-world scenarios; and (2) insufficient labeled data, which hampers the model's ability to learn complex cross-modal associations and limits generalization. To overcome these challenges, we propose a novel framework that leverages the semantic comprehension and reasoning capabilities of Large Language Models (LLMs). Specifically, for the mismatch issue, we employ LLMs to generate the text-image relevance score with inference reason to guide the subsequent modules. Then we design Text-image Relationship Predicting (TRP) module, which determines the final feature fusion weights based on the relevance score provided by LLMs. To mitigate data scarcity, we prompt LLMs to identify the key entities in text and incorporate them into the original input. Additionally, we design Text-image Relevance Features Learning (TRFL) module to construct positive and negative samples based on the relevance score, employing a supervised contrastive learning method to further enhance the model's ability to extract key features from image-text pairs. Experiments show that our proposed method achieves F1 scores of 75.32 % and 86.65 % on Twitter-2015 and Twitter-2017 datasets, respectively, demonstrating its effectiveness.
资助项目Beijing Natural Science Foundation[7244507]
WOS研究方向Computer Science
语种英语
WOS记录号WOS:001534525700012
出版者ELSEVIER
源URL[http://119.78.100.204/handle/2XEOYT63/41784]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Wan, Jing
作者单位1.Beijing Univ Chem Technol, Beijing 100029, Peoples R China
2.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab Network Data Sci & Technol, Beijing 100190, Peoples R China
3.China Acad Chinese Med Sci, Inst Basic Res Clin Med, Beijing 100700, Peoples R China
推荐引用方式
GB/T 7714		 Zeng, Qingyang,Yuan, Minghui,Su, Yueyang,et al. Improving multimodal named entity recognition via text-image relevance prediction with large language models[J]. NEUROCOMPUTING,2025,651:10.
APA Zeng, Qingyang,Yuan, Minghui,Su, Yueyang,Mi, Jia,Che, Qianzi,&Wan, Jing.(2025).Improving multimodal named entity recognition via text-image relevance prediction with large language models.NEUROCOMPUTING,651,10.
MLA Zeng, Qingyang,et al."Improving multimodal named entity recognition via text-image relevance prediction with large language models".NEUROCOMPUTING 651(2025):10.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。