中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Audio-guided self-supervised learning for disentangled visual speech representations

文献类型：期刊论文

作者Feng, Dalu1,2; Yang, Shuang1,2; Shan, Shiguang1,2; Chen, Xilin1,2
刊名FRONTIERS OF COMPUTER SCIENCE
出版日期2024-12-01
卷号18期号:6页码:3
ISSN号2095-2228
DOI10.1007/s11704-024-3787-8
英文摘要4 ConclusionIn this paper, we propose a novel two-branch framework to learn the disentangled visual speech representations based on two particular observations. Its main idea is to introduce the audio signal to guide the learning of speech-relevant cues and introduce a bottleneck to restrict the speech-irrelevant branch from learning high-frequency and fine-grained speech cues. Experiments on both the word-level and sentence-level audio-visual speech datasets LRW and LRS2-BBC show the effectiveness. Our future work is to explore more explicit auxiliary tasks and constraints beyond the reconstruction task of the speech-relevant and irrelevant branch to improve further its ability of capturing speech cues in the video. Meanwhile, it's also a nice try to combine multiple types of knowledge representations [10] to further boost the obtained speech epresentations, which is also left for the future work.
资助项目National Natural Science Foundation of China[62276247] ; National Natural Science Foundation of China[62076250]
WOS研究方向Computer Science
语种英语
WOS记录号WOS:001253372000003
出版者HIGHER EDUCATION PRESS
源URL[http://119.78.100.204/handle/2XEOYT63/39884]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Yang, Shuang
作者单位1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式
GB/T 7714		 Feng, Dalu,Yang, Shuang,Shan, Shiguang,et al. Audio-guided self-supervised learning for disentangled visual speech representations[J]. FRONTIERS OF COMPUTER SCIENCE,2024,18(6):3.
APA Feng, Dalu,Yang, Shuang,Shan, Shiguang,&Chen, Xilin.(2024).Audio-guided self-supervised learning for disentangled visual speech representations.FRONTIERS OF COMPUTER SCIENCE,18(6),3.
MLA Feng, Dalu,et al."Audio-guided self-supervised learning for disentangled visual speech representations".FRONTIERS OF COMPUTER SCIENCE 18.6(2024):3.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。