中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields

文献类型：期刊论文

作者Gao, Lin2,5; Liu, Feng-Lin2,5; Chen, Shu-Yu5; Jiang, Kaiwen1,5; Li, Chunpeng5; Lai, Yu-Kun4; Fu, Hongbo3
刊名ACM TRANSACTIONS ON GRAPHICS
出版日期2023-08-01
卷号42期号:4页码:17
ISSN号0730-0301
关键词Sketch-based Interaction Neural Radiance Fields Face Modeling Face Editing
DOI10.1145/3592100
英文摘要Realistic 3D facial generation based on Neural Radiance Fields (NeRFs) from 2D sketches benefits various applications. Despite the high realism of freeview rendering results of NeRFs, it is tedious and difficult for artists to achieve detailed 3D control and manipulation. Meanwhile, due to its conciseness and expressiveness, sketching has been widely used for 2D facial image generation and editing. Applying sketching to NeRFs is challenging due to the inherent uncertainty for 3D generation with 2D constraints, a significant gap in content richness when generating faces from sparse sketches, and potential inconsistencies for sequential multi-view editing given only 2D sketch inputs. To address these challenges, we present SketchFaceNeRF, a novel sketch-based 3D facial NeRF generation and editing method, to produce free-view photo-realistic images. To solve the challenge of sketch sparsity, we introduce a Sketch Tri-plane Prediction net to first inject the appearance into sketches, thus generating features given reference images to allow color and texture control. Such features are then lifted into compact 3D tri-planes to supplement the absent 3D information, which is important for improving robustness and faithfulness. However, during editing, consistency for unseen or unedited 3D regions is difficult to maintain due to limited spatial hints in sketches. We thus adopt a Mask Fusion module to transform free-view 2D masks (inferred from sketch editing operations) into the tri-plane space as 3D masks, which guide the fusion of the original and sketch-based generated faces to synthesize edited faces. We further design an optimization approach with a novel space loss to improve identity retention and editing faithfulness. Our pipeline enables users to flexibly manipulate faces from different viewpoints in 3D space, easily designing desirable facial models. Extensive experiments validate that our approach is superior to the state-of-the-art 2D sketch-based image generation and editing approaches in realism and faithfulness.
资助项目Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; National Natural Science Foundation of China[62061136007] ; National Natural Science Foundation of China[62102403] ; China Postdoctoral Science Foundation[2022M713205] ; Research Grants Council of the Hong Kong Special Administrative Region, China[CityU 11212119] ; Chow Sang Sang Group Research Fund[9229119] ; Centre for Applied Computing and Interactive Media (ACIM) of School of Creative Media, CityU
WOS研究方向Computer Science
语种英语
出版者ASSOC COMPUTING MACHINERY
WOS记录号WOS:001044671300125
源URL[http://119.78.100.204/handle/2XEOYT63/21391]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Gao, Lin
作者单位1.Beijing Jiaotong Univ, Beijing, Peoples R China
2.Univ Chinese Acad Sci, Beijing, Peoples R China
3.City Univ Hong Kong, Sch Creat Media, Hong Kong, Peoples R China
4.Cardiff Univ, Sch Comp Sci & Informat, Cardiff, Wales
5.Chinese Acad Sci, Beijing Key Lab Mobile Comp & Pervas Device, Inst Comp Technol, Beijing, Peoples R China
推荐引用方式
GB/T 7714		 Gao, Lin,Liu, Feng-Lin,Chen, Shu-Yu,et al. SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields[J]. ACM TRANSACTIONS ON GRAPHICS,2023,42(4):17.
APA Gao, Lin.,Liu, Feng-Lin.,Chen, Shu-Yu.,Jiang, Kaiwen.,Li, Chunpeng.,...&Fu, Hongbo.(2023).SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields.ACM TRANSACTIONS ON GRAPHICS,42(4),17.
MLA Gao, Lin,et al."SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields".ACM TRANSACTIONS ON GRAPHICS 42.4(2023):17.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。