3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction
文献类型：期刊论文
| 作者 | Chen, Shu-Yu4; Lai, Yu-Kun2; Xia, Shihong1,3; Rosin, Paul L.2; Gao, Lin1,3 |
| 刊名 | IEEE TRANSACTIONS ON MULTIMEDIA
 |
| 出版日期 | 2023 |
| 卷号 | 25页码:3166-3179 |
| ISSN号 | 1520-9210 |
| 关键词 | Communication eye tracking head-mounted display real-time facial performance capture user interaction virtual reality |
| DOI | 10.1109/TMM.2022.3156820 |
| 英文摘要 | With the rapid development of virtual reality (VR) technology, VR headsets, a.k.a. Head-Mounted Displays (HMDs), are widely available, allowing immersive 3D content to be viewed. A natural need for truly immersive VR is to allow bidirectional communication: the user should be able to interact with the virtual world using facial expressions and eye gaze, in addition to traditional means of interaction. The typical application scenario includes VR virtual conferencing and virtual roaming, where ideally users are able to see other users' expressions and have eye contact with them in the virtual world. In addition, eye gaze also provides a natural means of interaction with virtual objects. Despite significant achievements in recent years for reconstruction of 3D faces from RGB or RGB-D images, it remains a challenge to reliably capture and reconstruct 3D facial expressions including eye gaze when the user is wearing an HMD, because the majority of the face is occluded, especially those areas around the eyes which are essential for recognizing facial expressions and eye gaze. In this paper, we introduce a novel real-time system that is able to capture and reconstruct 3D faces wearing HMDs, and robustly recover eye gaze. We further propose a novel method to map eye gaze directions to the 3D virtual world, which provides a novel and useful interactive mode in VR. We compare our method with state-of-the-art techniques both qualitatively and quantitatively, and demonstrate the effectiveness of our system using live capture. |
| 资助项目 | National Natural Science Foundation of China[62102403] ; National Natural Science Foundation of China[61872440] ; National Natural Science Foundation of China[62061136007] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Science and Technology Service Network Initiative ; Chinese Academy of Sciences[KFJ-STS-QYZD- 2021-11-001] ; Youth Innovation Promotion Association CAS ; Royal Society Newton Advanced Fellowship[NAF\R2\192151] |
| WOS研究方向 | Computer Science ; Telecommunications |
| 语种 | 英语 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| WOS记录号 | WOS:001045742200017 |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/21373]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Gao, Lin |
| 作者单位 | 1.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 2.Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF24 4AG, Wales 3.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100190, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Chen, Shu-Yu,Lai, Yu-Kun,Xia, Shihong,et al. 3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2023,25:3166-3179. |
| APA | Chen, Shu-Yu,Lai, Yu-Kun,Xia, Shihong,Rosin, Paul L.,&Gao, Lin.(2023).3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction.IEEE TRANSACTIONS ON MULTIMEDIA,25,3166-3179. |
| MLA | Chen, Shu-Yu,et al."3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction".IEEE TRANSACTIONS ON MULTIMEDIA 25(2023):3166-3179. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。