Network Pruning for Bit-Serial Accelerators
文献类型：期刊论文
| 作者 | Zhao, Xiandong1; Wang, Ying2; Liu, Cheng2; Shi, Cong3; Tu, Kaijie1; Zhang, Lei1 |
| 刊名 | IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
 |
| 出版日期 | 2023-05-01 |
| 卷号 | 42期号:5页码:1597-1609 |
| ISSN号 | 0278-0070 |
| 关键词 | AI accelerators neural networks (NNs) NN compression |
| DOI | 10.1109/TCAD.2022.3203955 |
| 英文摘要 | Bit-serial architectures (BSAs) are becoming increasingly popular in low-power neural network processor (NNP) designs for edge scenarios. However, the performance and energy efficiency of state-of-the-art BSA NNPs heavily depends on both the proportion and distribution of ineffectual weight bits in neural networks (NNs). To boost the performance of typical BSA accelerators, we present Bit-Pruner, a software approach to learn BSA-favored NNs without resorting to hardware modifications. Bit-Pruner not only progressively prunes but also restructures the nonzero bits in weights so that the number of nonzero bits in the model can be reduced and the corresponding computing can be load-balanced to suit the target BSA accelerators. On top of Bit-Pruner, we further propose a Pareto frontier optimization algorithm to adjust the bit-pruning rate across network layers and fulfill diverse NN processing requirements in terms of performance and accuracy for various edge scenarios. However, an aggressive Bit-Pruner can lead to nontrivial accuracy loss, especially for lightweight NNs and complex tasks. To this end, the alternating direction method of multipliers (ADMMs) is adapted to the retraining phase in Bit-Pruner to smooth the abrupt disturbance due to bit-pruning and enhance the resulting model accuracy. According to the experiments, Bit-Pruner increases the bit-sparsity up to 94.4% with negligible accuracy degradation and achieves an optimized tradeoff between NN accuracy and energy efficiency even under very-aggressive performance constraints. When pruned models are deployed onto typical BSA accelerators, the average performance is 2.1 x and 1.6 x higher than the baseline networks without pruning and those with classical weight pruning, respectively. |
| 资助项目 | National Natural Science Foundation of China (NSFC)[61874124] ; National Natural Science Foundation of China (NSFC)[61876173] ; National Natural Science Foundation of China (NSFC)[62222411] ; National Natural Science Foundation of China (NSFC)[62174162] ; Zhejiang Lab[2021PC0AC01] ; State Key Laboratory of Processors ; Institute of Computing Technology ; CAS ; Beijing Key Laboratory of Mobile Computing and Pervasive Device ; Institute of Computing Technology ; Chinese Academy of Sciences |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| WOS记录号 | WOS:000976102300018 |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/21441]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Wang, Ying |
| 作者单位 | 1.Chinese Acad Sci, State Key Lab Processors, Beijing Key Lab Mobile Comp & Pervas Device, Inst Comp Technol,Inst Comp Technol,Univ Chinese A, Beijing 100089, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, State Key Lab Comp Architecture, Beijing 100089, Peoples R China 3.Chongqing Univ, Sch Microelect & Commun Engn, Chongqing 400044, Peoples R China |
| 推荐引用方式 GB/T 7714 | Zhao, Xiandong,Wang, Ying,Liu, Cheng,et al. Network Pruning for Bit-Serial Accelerators[J]. IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS,2023,42(5):1597-1609. |
| APA | Zhao, Xiandong,Wang, Ying,Liu, Cheng,Shi, Cong,Tu, Kaijie,&Zhang, Lei.(2023).Network Pruning for Bit-Serial Accelerators.IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS,42(5),1597-1609. |
| MLA | Zhao, Xiandong,et al."Network Pruning for Bit-Serial Accelerators".IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS 42.5(2023):1597-1609. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。