中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation

文献类型：期刊论文

作者Wu, Zhiyuan1; Sun, Sheng2; Wang, Yuwei2; Liu, Min2; Pan, Quyang2; Zhang, Junbo3,4; Li, Zeju5; Liu, Qingxiang6,7
刊名ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY
出版日期2024-04-01
卷号15期号:2页码:34
关键词Federated learning knowledge distillation proxy-data-free model heterogeneity
ISSN号2157-6904
DOI10.1145/3639369
英文摘要Federated learning (FL) is a privacy-preserving machine learning paradigm in which the server periodically aggregates localmodel parameters from cli entswithout assembling their private data. Constrained communication and personalization requirements pose severe challenges to FL. Federated distillation (FD) is proposed to simultaneously address the above two problems, which exchanges knowledge between the server and clients, supporting heterogeneous local models while significantly reducing communication overhead. However, most existing FD methods require a proxy dataset, which is often unavailable in reality. A few recent proxy-data-free FD approaches can eliminate the need for additional public data, but suffer from remarkable discrepancy among local knowledge due to client-side model heterogeneity, leading to ambiguous representation on the server and inevitable accuracy degradation. To tackle this issue, we propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC). FedDKC leverages well-designed refinement strategies to narrow local knowledge differences into an acceptable upper bound, so as to mitigate the negative effects of knowledge incongruence. Specifically, from perspectives of peak probability and Shannon entropy of local knowledge, we design kernel-based knowledge refinement (KKR) and searching-based knowledge refinement (SKR) respectively, and theoretically guarantee that the refined-local knowledge can satisfy an approximately-similar distribution and be regarded as congruent. Extensive experiments conducted on three common datasets demonstrate that our proposed FedDKC significantly outperforms the state-of-the-art on various heterogeneous settings while evidently improving the convergence speed.
资助项目National Key Research and Development Program of China[2021YFB2900102] ; National Natural Science Foundation of China[62072436] ; Innovation Capability Support Program of Shaanxi[2023-CX-TD-08] ; Shaanxi Qinchuangyuan scientists+engineers team[2023KXJ-040] ; Innovation Funding of ICT, CAS[E261080] ; Beijing Natural Science Foundation[4212021] ; Beijing Science and Technology Project[Z211100004121008]
WOS研究方向Computer Science
语种英语
WOS记录号WOS:001208775700009
出版者ASSOC COMPUTING MACHINERY
源URL[http://119.78.100.204/handle/2XEOYT63/39004]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Liu, Min; Pan, Quyang
作者单位1.China & Univ Chinese Acad Sci, Chinese Acad Sci, Inst Comp Technol, 6,Acad Sci South Rd, Beijing 100086, Peoples R China
2.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
3.JD Technol, JD iCity, Beijing, Peoples R China
4.JD Intelligent Cities Res, Beijing, Peoples R China
5.Beijing Univ Posts & Telecommun, Beijing, Peoples R China
6.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
7.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式
GB/T 7714		 Wu, Zhiyuan,Sun, Sheng,Wang, Yuwei,et al. Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation[J]. ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY,2024,15(2):34.
APA Wu, Zhiyuan.,Sun, Sheng.,Wang, Yuwei.,Liu, Min.,Pan, Quyang.,...&Liu, Qingxiang.(2024).Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation.ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY,15(2),34.
MLA Wu, Zhiyuan,et al."Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation".ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY 15.2(2024):34.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。