中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 多模态人工智能系统全国重点实验室
  5. 多媒体计算与图形学团队
Semantic Distance Adversarial Learning for Text-to-Image Synthesis

文献类型：期刊论文

作者Yuan, Bowen1; Sheng, Yefei1; Bao, Bing-Kun1,2; Chen, Yi-Ping Phoebe3; Xu, Changsheng2,4,5
刊名IEEE TRANSACTIONS ON MULTIMEDIA
出版日期2024
卷号26页码:1255-1266
关键词Text-to-image synthesis adversarial learning cycle consistency
ISSN号1520-9210
DOI10.1109/TMM.2023.3278992
通讯作者Bao, Bing-Kun(bingkunbao@njupt.edu.cn)
英文摘要Text-to-Image (T2I) synthesis is a cross-modality task that requires a text description as input to generate a realistic and semantically consistent image. To guarantee semantic consistency, previous studies regenerate text descriptions from synthetic images and align them with the given descriptions. However, the existing redescription modules lack explicit modeling of their training objectives, which is crucial for reliable measurement of semantic distance between redescriptions and given text inputs. Consequently, the aligned text redescriptions suffer from training bias caused by the emergence of adversarial image samples, unseen semantics, and mistaken contents from low-quality synthesized images. To this end, we propose a SEMantic distance Adversarial learning (SEMA) framework for Text-to-Image synthesis which strengthens semantic consistency from two aspects: 1) We introduce adversarial learning between the image generator and the text redescription module to mutually promote or demote the quality of generated image or text instances. This learning model ensures accurate redescription of image contents, thus diminishing the generation of adversarial image samples. 2) We introduce two-fold semantic distance discrimination (SEM distance) to characterize semantic relevance between matching text or image pairs. The unseen semantics and mistaken contents will be penalized with a large SEM distance. The proposed discrimination method also simplifies the model training process with no need to optimize multiple discriminators. Experimental results on CUB Birds 200 and MS-COCO datasets show that the proposed model outperforms the state-of-the-art methods.
资助项目National Key Research and Development Project
WOS研究方向Computer Science ; Telecommunications
语种英语
WOS记录号WOS:001166602700032
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
资助机构National Key Research and Development Project
源URL[http://ir.ia.ac.cn/handle/173211/57879]  
专题自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队
通讯作者Bao, Bing-Kun
作者单位1.Nanjing Univ Posts & Telecommun, Coll Telecommun & Informat Engn, Nanjing 210003, Peoples R China
2.Peng Cheng Lab, Shenzhen 518000, Peoples R China
3.La Trobe Univ, Melbourne, Vic 3086, Australia
4.Univ Chinese Acad Sci, Beijing 101408, Peoples R China
5.Chinese Acad Sci CASIA, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 101408, Peoples R China
推荐引用方式
GB/T 7714		 Yuan, Bowen,Sheng, Yefei,Bao, Bing-Kun,et al. Semantic Distance Adversarial Learning for Text-to-Image Synthesis[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:1255-1266.
APA Yuan, Bowen,Sheng, Yefei,Bao, Bing-Kun,Chen, Yi-Ping Phoebe,&Xu, Changsheng.(2024).Semantic Distance Adversarial Learning for Text-to-Image Synthesis.IEEE TRANSACTIONS ON MULTIMEDIA,26,1255-1266.
MLA Yuan, Bowen,et al."Semantic Distance Adversarial Learning for Text-to-Image Synthesis".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):1255-1266.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。