中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch

文献类型：期刊论文

作者Zhong, Qiaoling1; Zhang, Zhibin; Qiu, Qiang; Cheng, Xueqi
刊名IEEE ACCESS
出版日期2021
卷号9页码:51134-51145
关键词Network pruning inference acceleration model compression multiple GPUs
ISSN号2169-3536
DOI10.1109/ACCESS.2021.3065406
英文摘要Due to space and inference time restrictions, finding an efficient and sparse sub-network from a dense and over-parameterized network is critical for deploying neural networks on edge devices. Recent efforts explore obtaining a sparse sub-network by performing network pruning during training procedures to reduce training costs, such as memory and fioating-point operations (FLOPs). However, these works take more than 1.4 x the total number of iterations and try all possible pruning parameters manually to obtain sparse sub-networks. In this paper, we present a pruning framework Roulette to train a sparse network from scratch. First, we propose a novel method to train a sparse network by Pruning through the lens of Loss Landscape iteratively and automatically (PLL). We do a theoretical analysis that the curvature of the loss function is higher in the initial phase and can conduct us to start network pruning. According to our results on CIFAR-10/100 and ImageNet dataset, PLL saves up to 4x training FLOPs than prior works while maintaining comparable or even better accuracy. Then we design push and pull operations to synchronize the pruned weights on different GPUs during training, scaling PLL to multiple GPUs linearly. To our knowledge, Roulette is the first network pruning framework supporting multiple GPUs linearly.
资助项目Strategic Priority Research Program of Chinese Academy of Sciences (CAS)[XDA19020400]
WOS研究方向Computer Science ; Engineering ; Telecommunications
语种英语
WOS记录号WOS:000638386800001
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
源URL[http://119.78.100.204/handle/2XEOYT63/16767]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Zhong, Qiaoling
作者单位1.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab Network Data Sci & Technol, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Sch Comp & Control Engn, Beijing 100049, Peoples R China
推荐引用方式
GB/T 7714		 Zhong, Qiaoling,Zhang, Zhibin,Qiu, Qiang,et al. Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch[J]. IEEE ACCESS,2021,9:51134-51145.
APA Zhong, Qiaoling,Zhang, Zhibin,Qiu, Qiang,&Cheng, Xueqi.(2021).Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch.IEEE ACCESS,9,51134-51145.
MLA Zhong, Qiaoling,et al."Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch".IEEE ACCESS 9(2021):51134-51145.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。