中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
An adaptive joint optimization framework for pruning and quantization

文献类型：期刊论文

作者Li, Xiaohai1,2,3; Yang, Xiaodong1,2,3; Zhang, Yingwei1,2,3; Yang, Jianrong4,5; Chen, Yiqiang1,2,3
刊名INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS
出版日期2024-06-18
页码17
关键词Model compression Network pruning Quantization Mutual learning Multi-teacher knowledge distillation
ISSN号1868-8071
DOI10.1007/s13042-024-02229-w
英文摘要Pruning and quantization are among the most widely used techniques for deep learning model compression. Their combined application holds the potential for even greater performance gains. Most existing works combine pruning and quantization sequentially. However, this separation makes it difficult to fully leverage their complementarity and exploit the potential benefits of joint optimization. To address the limitations of existing methods, we propose A-JOPQ (adaptive joint optimization of pruning and quantization), an adaptive joint optimization framework for pruning and quantization. Starting with a deep neural network, A-JOPQ first constructs a pruning network through adaptive mutual learning with a quantization network. This process compensates for the loss of structural information during pruning. Subsequently, the pruning network is incrementally quantized using adaptive multi-teacher knowledge distillation of itself and the original uncompressed model. This approach effectively mitigates the adverse effects of quantization. Finally, A-JOPQ generates a pruning-quantization network that achieves significant model compression while maintaining high accuracy. Extensive experiments conducted on several public datasets demonstrate the superiority of our proposed method. Compared to existing methods, A-JOPQ achieves higher accuracy with a smaller model size. Additionally, we extend A-JOPQ to federated learning (FL) settings. Simulation experiments show that A-JOPQ can enhance FL by enabling resource-limited clients to participate effectively.
资助项目National Natural Science Foundation of China ; National Key Research and Development Plan of China[2021YFC2501202] ; Beijing Municipal Science & Technology Commission[Z221100002722009] ; Guangxi key research and development program[AB24010065] ; [82360569]
WOS研究方向Computer Science
语种英语
WOS记录号WOS:001249630500001
出版者SPRINGER HEIDELBERG
源URL[http://119.78.100.204/handle/2XEOYT63/39906]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Li, Xiaohai
作者单位1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
2.Beijing Key Lab Mobile Comp & Pervas Device, Beijing, Peoples R China
3.Univ Chinese Acad Sci, Beijing, Peoples R China
4.Guangxi Acad Med Sci, Peoples Hosp Guangxi Zhuang Autonomous Reg, Dept Hepatobiliary Pancreas & Spleen Surg, Nanning, Peoples R China
5.Peoples Hosp Guangxi Zhuang Autonomous Reg, Guangxi Clin Res Ctr Sleep Med, Nanning, Peoples R China
推荐引用方式
GB/T 7714		 Li, Xiaohai,Yang, Xiaodong,Zhang, Yingwei,et al. An adaptive joint optimization framework for pruning and quantization[J]. INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS,2024:17.
APA Li, Xiaohai,Yang, Xiaodong,Zhang, Yingwei,Yang, Jianrong,&Chen, Yiqiang.(2024).An adaptive joint optimization framework for pruning and quantization.INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS,17.
MLA Li, Xiaohai,et al."An adaptive joint optimization framework for pruning and quantization".INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS (2024):17.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。