中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising

文献类型：期刊论文

作者Li, Liang3; Cong, Gaoxiang2,3; Qi, Yuankai1; Zha, Zheng-Jun8; Wu, Qi7; Sheng, Quan Z.1; Huang, Qingming5,6; Yang, Ming-Hsuan4
刊名IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
出版日期2025-11-01
卷号47期号:11页码:10361-10377
关键词Videos Lips Visualization Acoustics Cloning Noise reduction Motion pictures Head Adaptation models Text to speech Visual voice cloning speech synthesis hierarchical phoneme modeling contrastive learning acoustic diffusion denoising
ISSN号0162-8828
DOI10.1109/TPAMI.2025.3597267
英文摘要Given a piece of text, a video clip, and reference audio, the movie dubbing (also known as Visual Voice Cloning, V2C) task aims to generate speeches that clone reference voice and align well with the video in both emotion and lip movement, which is more challenging than conventional text-to-speech synthesis tasks. To align the generated speech with the inherent lip motion of the given silent video, most existing works utilize each video frame to query textual phonemes. However, such an attention operation usually leads to mumble speech because different phonemes are fused for video frames corresponding to one phoneme (video frames are finer-grained than phonemes). To address this issue, we propose a diffusion-based movie dubbing architecture, which improves pronunciation by Hierarchical Phoneme Modeling (HPM) and generates better mel-spectrogram through Acoustic Diffusion Denoising (ADD). We term our model as HD-Dubber. Specifically, our HPM bridges the visual information and corresponding speech prosody from three aspects: (1) aligning lip movement with the speech duration based on each phoneme unit by contrastive learning; (2) conveying facial expression to phoneme-level energy and pitch; and (3) injecting global emotions captured from video scenes into prosody. On the other hand, ADD exploits a denoising diffusion framework to transform the noise signal into a mel-spectrogram via a parameterized Markov chain conditioned on textual phonemes and reference audio. ADD has two novel denoisers, the Style-adaptive Residual Denoiser (SRD) and the Phoneme-enhanced U-net Denoiser (PUD), to enhance speaker similarity and improve pronunciation quality. Extensive experimental results on the three benchmark datasets demonstrate the state-of-the-art performance of the proposed method. The source code and trained models will be made available to the public.
资助项目National Nature Science Foundation of China[62322211] ; National Nature Science Foundation of China[62236008] ; Pioneer and Leading Goose R&D Program of Zhejiang Province[2024C01023] ; Key Laboratory of Intelligent Processing Technology for Digital Music (Zhejiang Conservatory of Music), Ministry of Culture and Tourism[2023DMKLB004]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:001587252700038
出版者IEEE COMPUTER SOC
源URL[http://119.78.100.204/handle/2XEOYT63/41624]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Cong, Gaoxiang; Qi, Yuankai
作者单位1.Macquarie Univ, Sch Comp, Sydney, NSW 2113, Australia
2.Univ Chinese Acad Sci, Beijing 101408, Peoples R China
3.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
4.Univ Calif Merced, EECS, Merced, CA 95344 USA
5.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
6.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China
7.Univ Adelaide, Australian Inst Machine Learning, Sch Comp Sci, Adelaide, SA 5005, Australia
8.Univ Sci & Technol China, Hefei 230052, Peoples R China
推荐引用方式
GB/T 7714		 Li, Liang,Cong, Gaoxiang,Qi, Yuankai,et al. Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2025,47(11):10361-10377.
APA Li, Liang.,Cong, Gaoxiang.,Qi, Yuankai.,Zha, Zheng-Jun.,Wu, Qi.,...&Yang, Ming-Hsuan.(2025).Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,47(11),10361-10377.
MLA Li, Liang,et al."Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 47.11(2025):10361-10377.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。