中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
Robust Pose Transfer With Dynamic Details Using Neural Video Rendering

文献类型：期刊论文

作者Sun, Yang-Tian4,5; Huang, Hao-Zhi3; Wang, Xuan2; Lai, Yu-Kun1; Liu, Wei2; Gao, Lin4,5
刊名IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
出版日期2023-02-01
卷号45期号:2页码:2660-2666
关键词Deep generative model dynamic details generation human video synthesis neural rendering pose transfer
ISSN号0162-8828
DOI10.1109/TPAMI.2022.3166989
英文摘要Pose transfer of human videos aims to generate a high-fidelity video of a target person imitating actions of a source person. A few studies have made great progress either through image translation with deep latent features or neural rendering with explicit 3D features. However, both of them rely on large amounts of training data to generate realistic results, and the performance degrades on more accessible Internet videos due to insufficient training frames. In this paper, we demonstrate that the dynamic details can be preserved even when trained from short monocular videos. Overall, we propose a neural video rendering framework coupled with an image-translation-based dynamic details generation network (D-2 G-Net), which fully utilizes both the stability of explicit 3D features and the capacity of learning components. To be specific, a novel hybrid texture representation is presented to encode both the static and pose-varying appearance characteristics, which is then mapped to the image space and rendered as a detail-rich frame in the neural rendering stage. Through extensive comparisons, we demonstrate that our neural human video renderer is capable of achieving both clearer dynamic details and more robust performance even on accessible short videos with only 2 k similar to 4 k frames, as illustrated in Fig. 1.
资助项目Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; National Natural Science Foundation of China[62061136007] ; National Natural Science Foundation of China[61872440] ; Royal Society Newton Advanced Fellowship[NAF\R2\192151] ; Tencent AI Lab Rhino-Bird Focused Research Program ; Youth Innovation Promotion Association CAS
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:000912386000086
出版者IEEE COMPUTER SOC
源URL[http://119.78.100.204/handle/2XEOYT63/20021]  
专题中国科学院计算技术研究所期刊论文
通讯作者Gao, Lin
作者单位1.Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF10 3AT, Wales
2.Tencent, Shenzhen 518054, Guangdong, Peoples R China
3.Xverse, Shenzhen 518100, Guangdong, Peoples R China
4.Univ Chinese Acad Sci, Beijing 101408, Peoples R China
5.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100045, Peoples R China
推荐引用方式
GB/T 7714		 Sun, Yang-Tian,Huang, Hao-Zhi,Wang, Xuan,et al. Robust Pose Transfer With Dynamic Details Using Neural Video Rendering[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(2):2660-2666.
APA Sun, Yang-Tian,Huang, Hao-Zhi,Wang, Xuan,Lai, Yu-Kun,Liu, Wei,&Gao, Lin.(2023).Robust Pose Transfer With Dynamic Details Using Neural Video Rendering.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(2),2660-2666.
MLA Sun, Yang-Tian,et al."Robust Pose Transfer With Dynamic Details Using Neural Video Rendering".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.2(2023):2660-2666.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。