中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective

文献类型：期刊论文

作者Kong, Xiaoyu6,7; Deng, Yingying5; Tang, Fan4; Dong, Weiming5; Ma, Chongyang3; Chen, Yongyong; He, Zhenyu1,2; Xu, Changsheng5
刊名IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
出版日期2023-01-06
页码15
ISSN号2162-237X
关键词Correlation Task analysis Optical imaging Integrated optics Lighting Optical fiber networks Image reconstruction Arbitrary stylization channel correlation cross-domain feature migration
DOI10.1109/TNNLS.2022.3230084
英文摘要Arbitrary image stylization by neural networks has become a popular topic, and video stylization is attracting more attention as an extension of image stylization. However, when image stylization methods are applied to videos, unsatisfactory results that suffer from severe flickering effects appear. In this article, we conducted a detailed and comprehensive analysis of the cause of such flickering effects. Systematic comparisons among typical neural style transfer approaches show that the feature migration modules for state-of-the-art (SOTA) learning systems are ill-conditioned and could lead to a channelwise misalignment between the input content representations and the generated frames. Unlike traditional methods that relieve the misalignment via additional optical flow constraints or regularization modules, we focus on keeping the temporal consistency by aligning each output frame with the input frame. To this end, we propose a simple yet efficient multichannel correlation network (MCCNet), to ensure that output frames are directly aligned with inputs in the hidden feature space while maintaining the desired style patterns. An inner channel similarity loss is adopted to eliminate side effects caused by the absence of nonlinear operations such as softmax for strict alignment. Furthermore, to improve the performance of MCCNet under complex light conditions, we introduce an illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks.
资助项目National Key R&D Program of China[2020AAA0106200] ; National Science Foundation of China[U20B2070] ; National Science Foundation of China[61832016] ; National Science Foundation of China[62102162] ; National Science Foundation of China[62172126] ; National Science Foundation of China[6216063] ; Major Key Project of Peng Cheng Laboratory[PCL2021A03-1] ; Shenzhen Research Council[JCYJ20210324120202006] ; Guangdong Natural Science Foundation[2022A1515010819]
WOS研究方向Computer Science ; Engineering
语种英语
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号WOS:000915636700001
源URL[http://119.78.100.204/handle/2XEOYT63/19973]  
专题中国科学院计算技术研究所期刊论文
通讯作者Tang, Fan
作者单位1.Peng Cheng Lab, Shenzhen 518055, Peoples R China
2.Harbin Inst Technol, Dept Comp Sci, Shenzhen 518073, Peoples R China
3.Kuaishou Technol, Beijing 100085, Peoples R China
4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
5.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
6.Harbin Inst Technol, Sch Comp Sci & Technol, Shenzhen 518073, Peoples R China
7.Jilin Univ, Sch Artificial Intelligence, Changchun 130012, Peoples R China
推荐引用方式
GB/T 7714		 Kong, Xiaoyu,Deng, Yingying,Tang, Fan,et al. Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2023:15.
APA Kong, Xiaoyu.,Deng, Yingying.,Tang, Fan.,Dong, Weiming.,Ma, Chongyang.,...&Xu, Changsheng.(2023).Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,15.
MLA Kong, Xiaoyu,et al."Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2023):15.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。