中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
BIRD plus : Design of a Lightweight Communication Compressor for Resource-Constrained Distribution Learning Platforms

文献类型：期刊论文

作者Wu, Donglei2,3; Yang, Weihao2,3; Zou, Xiangyu2,3; Feng, Hao1; Tao, Dingwen4; Li, Shiyi2,3; Xia, Wen2,3; Fang, Binxing2,3
刊名IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
出版日期2024-11-01
卷号35期号:11页码:2193-2207
关键词Indexes Costs Computational modeling Distance learning Computer aided instruction Training Tensors Distributed learning communication compression random sampling neural network
ISSN号1045-9219
DOI10.1109/TPDS.2024.3447221
英文摘要The Top-K sparsification-based compression framework is extensively explored for reducing communication costs in distributed learning. However, we identified several issues with existing Top-K sparsification-based compression methods: (i) The limited compressibility of the Top-K parameter's indexes critically restricts the overall communication compression ratio; (ii) Several time-consuming compression operations significantly offset the benefits of communication compression; (iii) The use of error feedback techniques to maintain model quality results in a high memory footprint consumption. To solve these issues, we propose BIRD, a lightweight tensor-wise Bi-Random sampling strategy with an expectation invariance property. Specifically, BIRD applies a tensor-wise index sharing mechanism that reduces the index proportion by allowing multiple tensor elements to share a single index, thus improving the overall compression ratio. Additionally, BIRD replaces the time-consuming Top-K sorting with a faster Bi-Random sampling strategy based on the aforementioned index sharing mechanism, significantly reducing compression overheads; Moreover, BIRD establishes an expectation invariance property into the Bi-Random sampling to ensure an approximate unbiased representation for the $L_1$L1-norm of the sampled tensors, effectively maintaining the model quality without incurring extra memory costs. We further optimize BIRD to BIRD+ by introducing the uniform distribution-based sampling and Gamma correction on the tensor-wise sampling process, achieving a more flexibly adjustment of the sparsity with better convergence performance. Experimental evaluations across multiple conventional distributed learning tasks demonstrate that compared to state-of-the-art approaches, BIRD+ achieves higher communication compression ratios up to 36.2x and higher computation throughput up to 149.6x while maintaining the model quality without incurring extra memory costs.
资助项目Major Key Project of PCL[PCL2022A03] ; Shenzhen Science and Technology Program[RCYX20210609104510007] ; Shenzhen Science and Technology Program[KJZD20230923114610021] ; Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies[2022B1212010005] ; Guangdong Basic and Applied Basic Research Foundation[2023A1515110072] ; National Natural Science Foundation of China[62472127] ; National Natural Science Foundation of China[62032023] ; National Natural Science Foundation of China[T2125013] ; Innovation Funding of ICT, CAS[E461050]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:001320540600003
出版者IEEE COMPUTER SOC
源URL[http://119.78.100.204/handle/2XEOYT63/39537]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Xia, Wen
作者单位1.Indiana Univ, Bloomington, IN 47405 USA
2.Harbin Inst Technol, Guangdong Prov Key Lab Novel Secur Intelligence Te, Shenzhen 518055, Peoples R China
3.Peng Cheng Lab, Dept New Networks, Shenzhen 518055, Peoples R China
4.Chinese Acad Sci, Inst Comp Technol, State Key Lab Processors, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714		 Wu, Donglei,Yang, Weihao,Zou, Xiangyu,et al. BIRD plus : Design of a Lightweight Communication Compressor for Resource-Constrained Distribution Learning Platforms[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2024,35(11):2193-2207.
APA Wu, Donglei.,Yang, Weihao.,Zou, Xiangyu.,Feng, Hao.,Tao, Dingwen.,...&Fang, Binxing.(2024).BIRD plus : Design of a Lightweight Communication Compressor for Resource-Constrained Distribution Learning Platforms.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,35(11),2193-2207.
MLA Wu, Donglei,et al."BIRD plus : Design of a Lightweight Communication Compressor for Resource-Constrained Distribution Learning Platforms".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 35.11(2024):2193-2207.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。