Rethinking the Importance of Quantization Bias, Toward Full Low-Bit Training
文献类型：期刊论文
| 作者 | Liu, Chang2,3,4; Zhang, Xishan3,4; Zhang, Rui3,4; Li, Ling1,2; Zhou, Shiyi3; Huang, Di2,3,4; Li, Zhen3; Du, Zidong3,4; Liu, Shaoli3; Chen, Tianshi3 |
| 刊名 | IEEE TRANSACTIONS ON IMAGE PROCESSING
 |
| 出版日期 | 2022 |
| 卷号 | 31页码:7006-7019 |
| 关键词 | Neural network acceleration low precision training quantization |
| ISSN号 | 1057-7149 |
| DOI | 10.1109/TIP.2022.3216776 |
| 英文摘要 | Quantization is a promising technique to reduce the computation and storage costs of DNNs. Low-bit ( $\leq8$ bits) precision training remains an open problem due to the difficulty of gradient quantization. In this paper, we find two long-standing misunderstandings of the bias of gradient quantization noise. First, the large bias of gradient quantization noise, instead of the variance, is the key factor of training accuracy loss. Second, the widely used stochastic rounding cannot solve the training crash problem caused by the gradient quantization bias in practice. Moreover, we find that the asymmetric distribution of gradients causes a large bias of gradient quantization noise. Based on our findings, we propose a novel adaptive piecewise quantization method to effectively limit the bias of gradient quantization noise. Accordingly, we propose a new data format, Piecewise Fixed Point (PWF), to present data after quantization. We apply our method to different applications including image classification, machine translation, optical character recognition, and text classification. We achieve approximately $1.9\sim 3.5\times $ speedup compared with full precision training with an accuracy loss of less than 0.5%. To the best of our knowledge, this is the first work to quantize gradients of all layers to 8 bits in both large-scale CNN and RNN training with negligible accuracy loss. |
| 资助项目 | National Key Research and Development Programof China[2017YFA0700902] ; National Key Research and Development Programof China[2017YFA0700903] ; NSF of China[61925208] ; NSF of China[61906179] ; NSF of China[62102399] ; NSF of China[61732020] ; NSF of China[U19B2019] ; Strategic Priority Research Program of Chinese Academy of Science[XDB32050200] ; Beijing Academy of Artificial Intelligence(BAAI) ; Beijing Nova Program of Science and Technology[Z191100001119093] ; CAS Project for Young Scientistsin Basic Research[YSBR-029] ; Youth InnovationPromotion Association |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:000888975000003 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/20282]  |
| 专题 | 中国科学院计算技术研究所期刊论文 |
| 通讯作者 | Li, Ling |
| 作者单位 | 1.Chinese Acad Sci, Inst Software, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Cambricon Technol, Beijing 100191, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, SKL Comp Architecture, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Liu, Chang,Zhang, Xishan,Zhang, Rui,et al. Rethinking the Importance of Quantization Bias, Toward Full Low-Bit Training[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2022,31:7006-7019. |
| APA | Liu, Chang.,Zhang, Xishan.,Zhang, Rui.,Li, Ling.,Zhou, Shiyi.,...&Chen, Tianshi.(2022).Rethinking the Importance of Quantization Bias, Toward Full Low-Bit Training.IEEE TRANSACTIONS ON IMAGE PROCESSING,31,7006-7019. |
| MLA | Liu, Chang,et al."Rethinking the Importance of Quantization Bias, Toward Full Low-Bit Training".IEEE TRANSACTIONS ON IMAGE PROCESSING 31(2022):7006-7019. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。