BitNet: 1-bit Pre-training for Large Language Models
文献类型：期刊论文
| 作者 | Wang, Hongyu4,5; Ma, Shuming3; Ma, Lingxiao3; Wang, Lei1; Wang, Wenhui3; Dong, Li3; Huang, Shaohan3; Wang, Huaijie2; Xue, Jilong3; Wang, Ruiping4,5 |
| 刊名 | JOURNAL OF MACHINE LEARNING RESEARCH
 |
| 出版日期 | 2025 |
| 卷号 | 26页码:29 |
| 关键词 | Natural Language Processing Large Language Models 1-bit Pre-training Efficiency Model Architecture |
| ISSN号 | 1532-4435 |
| 英文摘要 | The increasing size of large language models (LLMs) has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. Previous research typically applies quantization after pre-training. While these methods avoid the need for model retraining, they often cause notable accuracy loss at extremely low bit-widths. In this work, we explore the feasibility and scalability of 1-bit pre-training. We introduce BitNet b1 and BitNet b1.58, the scalable and stable 1-bit Transformer architecture designed for LLMs. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results show that BitNet b1 achieves competitive performance, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. With the ternary weight, BitNet b1.58 matches the half-precision Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, BitNet defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. It enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs. |
| WOS研究方向 | Automation & Control Systems ; Computer Science |
| 语种 | 英语 |
| WOS记录号 | WOS:001565772300001 |
| 出版者 | MICROTOME PUBL |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/41744]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Wang, Hongyu |
| 作者单位 | 1.Univ Chinese Acad Sci, Beijing, Peoples R China 2.Tsinghua Univ, Inst Interdisciplinary Informat Sci, Beijing, Peoples R China 3.Microsoft Res, Silverdale, WA USA 4.Univ Chinese Acad Sci, Beijing, Peoples R China 5.Chinese Acad Sci, Inst Comp Technol, Key Lab AI Safety Chinese Acad Sci CAS, Beijing, Peoples R China |
| 推荐引用方式 GB/T 7714 | Wang, Hongyu,Ma, Shuming,Ma, Lingxiao,et al. BitNet: 1-bit Pre-training for Large Language Models[J]. JOURNAL OF MACHINE LEARNING RESEARCH,2025,26:29. |
| APA | Wang, Hongyu.,Ma, Shuming.,Ma, Lingxiao.,Wang, Lei.,Wang, Wenhui.,...&Wei, Furu.(2025).BitNet: 1-bit Pre-training for Large Language Models.JOURNAL OF MACHINE LEARNING RESEARCH,26,29. |
| MLA | Wang, Hongyu,et al."BitNet: 1-bit Pre-training for Large Language Models".JOURNAL OF MACHINE LEARNING RESEARCH 26(2025):29. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。