Cross Modal Compression With Variable Rate Prompt
文献类型：期刊论文
| 作者 | Gao, Junlong4; Li, Jiguo2,3; Jia, Chuanmin1; Wang, Shanshe4; Ma, Siwei4; Gao, Wen4 |
| 刊名 | IEEE TRANSACTIONS ON MULTIMEDIA
 |
| 出版日期 | 2024 |
| 卷号 | 26页码:3444-3456 |
| 关键词 | Cross modal compression semantic fidelity variable rate prompt |
| ISSN号 | 1520-9210 |
| DOI | 10.1109/TMM.2023.3310657 |
| 英文摘要 | Traditional image/video compression compresses the highly redundant visual data while preserving signal fidelity. Recently, cross modal compression (CMC) is proposed to compress the data into a compact, human-comprehensible domain (such as text) with an ultra-high compression ratio while preserving semantic fidelity for machine analysis and semantic monitoring. CMC is with a constant rate because the CMC encoder can only represent the data with a fixed grain. But in practice, variable rate is necessary due to the complicated and dynamic transmission conditions, the different storage mediums, and the diverse levels of application requirements. To deal with this problem, in this paper, we propose variable rate cross modal compression (VR-CMC), where we introduce variable rate prompt to represent the data with different grains. Variable rate prompt is composed of three strategies. Specifically, 1) target length prompt (TLP) introduces the target length into the language prompt to guide the generation of the text representation; 2) decaying EOS probability (DEP) exponentially decays the probability of the EOS token with regard to the decoding step and target length, where the EOS (end-of-sequence) token is a special token indicating the end of the text; 3) text augmentation (TA) enriches the training data and makes the text representation length more balanced when training. Experimental results show that our proposed VR-CMC can effectively control the rate in the CMC framework and achieve state-of-the-art performance on MSCOCO and IM2P datasets. |
| 资助项目 | National Natural Science Foundation of China |
| WOS研究方向 | Computer Science ; Telecommunications |
| 语种 | 英语 |
| WOS记录号 | WOS:001165348200028 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/38738]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Jia, Chuanmin |
| 作者单位 | 1.Peking Univ, Wangxuan Inst Comp Technol, Beijing 100871, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 4.Peking Univ, Natl Engn Res Ctr Visual Technol, Sch Comp Sci, Beijing 100871, Peoples R China |
| 推荐引用方式 GB/T 7714 | Gao, Junlong,Li, Jiguo,Jia, Chuanmin,et al. Cross Modal Compression With Variable Rate Prompt[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:3444-3456. |
| APA | Gao, Junlong,Li, Jiguo,Jia, Chuanmin,Wang, Shanshe,Ma, Siwei,&Gao, Wen.(2024).Cross Modal Compression With Variable Rate Prompt.IEEE TRANSACTIONS ON MULTIMEDIA,26,3444-3456. |
| MLA | Gao, Junlong,et al."Cross Modal Compression With Variable Rate Prompt".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):3444-3456. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。