中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Fast and accurate variable batch size convolution neural network training on large scale distributed systems

文献类型：期刊论文

作者Hu, Zhongzhe1,2; Xiao, Junmin1; Sun, Ninghui1; Tan, Guangming1
刊名CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE
出版日期2022-06-06
页码26
ISSN号1532-0626
关键词deep learning distributed computing ImageNet-1K large-batch training synchronous SGD
DOI10.1002/cpe.7119
英文摘要Large-scale distributed convolution neural network (CNN) training brings two performance challenges: model performance and system performance. Large batch size usually leads to model test accuracy loss, which counteracts the benefits of parallel SGD. The existing solutions require massive hyperparameter hand-tuning. To overcome this difficult, we analyze the training process and find that earlier training stages are more sensitive to batch size. Accordingly, we assert that different stages should use different batch size, and propose a variable batch size strategy. In order to remain high test accuracy under larger batch size cases, we design an auto-tuning engine for automatic parameter tuning in the proposed variable batch size strategy. Furthermore, we develop a dataflow implementation approach to achieve the high-throughput CNN training on supercomputer system. Our approach has achieved high generalization performance on SOAT CNN networks. For the ShuffleNet, ResNet-50, and ResNet-101 training with ImageNet-1K dataset, we scale the batch size to 120 K without accuracy loss and to 128 K with only a slight loss. And the dataflow implementation approach achieves 93.5% scaling efficiency on 1024 GPUs compared with the state-of-the-art.
WOS研究方向Computer Science
语种英语
出版者WILEY
WOS记录号WOS:000806476800001
源URL[http://119.78.100.204/handle/2XEOYT63/19600]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Hu, Zhongzhe
作者单位1.Chinese Acad Sci, Inst Comp Technol, 6 Kexueyuan South Rd, Beijing, Peoples R China
2.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式
GB/T 7714		 Hu, Zhongzhe,Xiao, Junmin,Sun, Ninghui,et al. Fast and accurate variable batch size convolution neural network training on large scale distributed systems[J]. CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE,2022:26.
APA Hu, Zhongzhe,Xiao, Junmin,Sun, Ninghui,&Tan, Guangming.(2022).Fast and accurate variable batch size convolution neural network training on large scale distributed systems.CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE,26.
MLA Hu, Zhongzhe,et al."Fast and accurate variable batch size convolution neural network training on large scale distributed systems".CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE (2022):26.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。