中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
Learning Semantic Concepts and Order for Image and Sentence Matching

文献类型:会议论文

AuthorHuang, Yan; Wu, Qi; Wang, Liang
Issued Date2018-06
Conference Date2018.6.18-2018.6.22
Conference PlaceSalt Lake City
KeywordImage And Sentence Matching
Volume0
Issue0
DOI0
Pages6163-6171
English Abstract

Image and sentence matching has made great progress recently, but it remains challenging due to the large visual semantic discrepancy. This mainly arises from that the representation of pixel-level image usually lacks of high-level semantic information as in its matched sentence. In this work, we propose a semantic-enhanced image and sentence matching model, which can improve the image representation by learning semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its semantic concepts, including objects, properties, actions, etc. Then, considering that different orders of semantic concepts lead to diverse semantic meanings, we use a context-gated sentence generation scheme for semantic order learning. It simultaneously uses the image global context containing concept relations as reference and the groundtruth semantic order in the matched sentence as supervision. After obtaining the improved image representation, we learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate the effectiveness of our learned semantic concepts and order, by achieving the state-of-the-art results on two public benchmark datasets.

Author of SourceMichael Brown
PublisherIEEE
Publish PlaceUSA
Language英语
URL查看原文
源URL[http://ir.ia.ac.cn/handle/173211/25799]  
Collection自动化研究所_智能感知与计算研究中心
Affiliation中科院自动化所
Recommended Citation
GB/T 7714
Huang, Yan,Wu, Qi,Wang, Liang. Learning Semantic Concepts and Order for Image and Sentence Matching[C]. 见:. Salt Lake City. 2018.6.18-2018.6.22.

入库方式: OAI收割

来源:自动化研究所

浏览0
下载0
收藏0
其他版本

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.