Aligning Logits Generatively for Principled Black-Box Knowledge Distillation in the Wild
文献类型：期刊论文
| 作者 | Xiang, Xiang1,2,3; Ma, Jing1; Wu, Dongrui1; Zeng, Zhigang1; Chen, Xilin4 |
| 刊名 | IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
 |
| 出版日期 | 2025-12-01 |
| 卷号 | 47期号:12页码:11929-11945 |
| 关键词 | Data models Adaptation models Cloud computing Training Predictive models Image edge detection Generators Computational modeling Servers Model compression Cloud-to-edge model compression knowledge distillation generative adversarial network test-time adaptation |
| ISSN号 | 0162-8828 |
| DOI | 10.1109/TPAMI.2025.3602663 |
| 英文摘要 | Black-Box Knowledge Distillation (B2KD) is a conservative task in cloud-to-edge model compression, emphasizing the protection of data privacy and model copyrights on both the cloud and edge. With invisible data and models hosted on the server, B2KD aims to utilize only the API queries of the teacher model's inference results in the cloud to effectively distill a lightweight student model deployed on edge devices. B2KD faces challenges such as limited Internet exchange and edge-cloud disparity in data distribution. To address these issues, we theoretically provide a new optimization direction from logits to cell boundary, different from direct logits alignment, and formalize a workflow comprising deprivatization, distillation, and adaptation at test time. Guided by this, we propose a method, Mapping-Emulation KD (MEKD), to enhance the robust prediction and anti-interference capabilities of the student model on edge devices for any unknown data distribution in real-world scenarios. Our method does not differentiate between treating soft or hard responses and consists of: 1) deprivatization: emulating the inverse mapping of the teacher function with a generator, 2) distillation: aligning low-dimensional logits of the teacher and student models by reducing the distance of high-dimensional image points, and 3) adaptation: correcting the student's online prediction bias through a graph propagation-based only-forward test-time adaptation algorithm. Our method demonstrates inspiring performance for edge model distillation and adaptation across different teacher-student pairs. We validate the effectiveness of our method on multiple image recognition benchmarks and various Deep Neural Network models, achieving state-of-the-art performance and showcasing its practical value in remote sensing image recognition applications. |
| 资助项目 | Foundation for Outstanding Research Groups of Hubei Province[2025AFA012] ; The 111 Project on Computational Intelligence and Intelligent Control[B18024] ; Project of Peng Cheng Lab[PCL2025AS214] ; Natural Science Fund of Hubei Province[2022CFB823] |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:001609560700017 |
| 出版者 | IEEE COMPUTER SOC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/42909]  |
| 专题 | 中国科学院计算技术研究所 |
| 通讯作者 | Xiang, Xiang |
| 作者单位 | 1.Huazhong Univ Sci & Technol, Sch Artificial Intelligence & Automat, Wuhan 430074, Peoples R China 2.Huazhong Univ Sci & Technol, Sch Comp Sci & Technol, Wuhan 430074, Peoples R China 3.Peng Cheng Lab, Shenzhen 51800, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Xiang, Xiang,Ma, Jing,Wu, Dongrui,et al. Aligning Logits Generatively for Principled Black-Box Knowledge Distillation in the Wild[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2025,47(12):11929-11945. |
| APA | Xiang, Xiang,Ma, Jing,Wu, Dongrui,Zeng, Zhigang,&Chen, Xilin.(2025).Aligning Logits Generatively for Principled Black-Box Knowledge Distillation in the Wild.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,47(12),11929-11945. |
| MLA | Xiang, Xiang,et al."Aligning Logits Generatively for Principled Black-Box Knowledge Distillation in the Wild".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 47.12(2025):11929-11945. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。