Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization
文献类型：期刊论文
| 作者 | Deng, Lei4; Wu, Yujie4; Hu, Yifan4; Liang, Ling3; Li, Guoqi4; Hu, Xing2; Ding, Yufei1; Li, Peng3; Xie, Yuan3 |
| 刊名 | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
 |
| 出版日期 | 2021-10-29 |
| 页码 | 15 |
| ISSN号 | 2162-237X |
| 关键词 | Neurons Computational modeling Quantization (signal) Optimization Encoding Task analysis Synapses Activity regularization alternating direction method of multiplier (ADMM) connection pruning spiking neural network (SNN) compression weight quantization |
| DOI | 10.1109/TNNLS.2021.3109064 |
| 英文摘要 | As well known, the huge memory and compute costs of both artificial neural networks (ANNs) and spiking neural networks (SNNs) greatly hinder their deployment on edge devices with high efficiency. Model compression has been proposed as a promising technique to improve the running efficiency via parameter and operation reduction, whereas this technique is mainly practiced in ANNs rather than SNNs. It is interesting to answer how much an SNN model can be compressed without compromising its functionality, where two challenges should be addressed: 1) the accuracy of SNNs is usually sensitive to model compression, which requires an accurate compression methodology and 2) the computation of SNNs is event-driven rather than static, which produces an extra compression dimension on dynamic spikes. To this end, we realize a comprehensive SNN compression through three steps. First, we formulate the connection pruning and weight quantization as a constrained optimization problem. Second, we combine spatiotemporal backpropagation (STBP) and alternating direction method of multipliers (ADMMs) to solve the problem with minimum accuracy loss. Third, we further propose activity regularization to reduce the spike events for fewer active operations. These methods can be applied in either a single way for moderate compression or a joint way for aggressive compression. We define several quantitative metrics to evaluate the compression performance for SNNs. Our methodology is validated in pattern recognition tasks over MNIST, N-MNIST, CIFAR10, and CIFAR100 datasets, where extensive comparisons, analyses, and insights are provided. To the best of our knowledge, this is the first work that studies SNN compression in a comprehensive manner by exploiting all compressible components and achieves better results. |
| 资助项目 | National Key Research and Development Program of China[2018AAA0102600] ; National Key Research and Development Program of China[2018YEF0200200] ; National Natural Science Foundation of China[61876215] ; Beijing Academy of Artificial Intelligence (BAAI) ; Science and Technology Major Project of Guangzhou[202007030006] ; Open Project of Zhejiang Laboratory ; Institute for Guo Qiang of Tsinghua University |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| WOS记录号 | WOS:000733540300001 |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/17919]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Li, Guoqi; Hu, Xing |
| 作者单位 | 1.Univ Calif Santa Barbara, Dept Comp Sci, Santa Barbara, CA 93106 USA 2.Chinese Acad Sci, Inst Comp Technol, State Key Lab Comp Architecture, Beijing 100190, Peoples R China 3.Univ Calif Santa Barbara, Dept Elect & Comp Engn, Santa Barbara, CA 93106 USA 4.Tsinghua Univ, Dept Precis Instrument, Ctr Brain Inspired Comp Res, Beijing 100084, Peoples R China |
| 推荐引用方式 GB/T 7714 | Deng, Lei,Wu, Yujie,Hu, Yifan,et al. Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2021:15. |
| APA | Deng, Lei.,Wu, Yujie.,Hu, Yifan.,Liang, Ling.,Li, Guoqi.,...&Xie, Yuan.(2021).Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,15. |
| MLA | Deng, Lei,et al."Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2021):15. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。