Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation
文献类型：期刊论文
| 作者 | Zimo Yin; Jian Pu; Yijie Zhou; Xiangyang Xue |
| 刊名 | IEEE/CAA Journal of Automatica Sinica
 |
| 出版日期 | 2024 |
| 卷号 | 11期号:11页码:2270-2283 |
| 关键词 | Cluster-based regularization iterative prediction refinement model-agnostic framework self-knowledge distillation (SKD) two-stage knowledge transfer |
| ISSN号 | 2329-9266 |
| DOI | 10.1109/JAS.2024.124629 |
| 英文摘要 | Knowledge distillation (KD) enhances student network generalization by transferring dark knowledge from a complex teacher network. To optimize computational expenditure and memory utilization, self-knowledge distillation (SKD) extracts dark knowledge from the model itself rather than an external teacher network. However, previous SKD methods performed distillation indiscriminately on full datasets, overlooking the analysis of representative samples. In this work, we present a novel two-stage approach to providing targeted knowledge on specific samples, named two-stage approach self-knowledge distillation (TOAST). We first soften the hard targets using class medoids generated based on logit vectors per class. Then, we iteratively distill the under-trained data with past predictions of half the batch size. The two-stage knowledge is linearly combined, efficiently enhancing model performance. Extensive experiments conducted on five backbone architectures show our method is model-agnostic and achieves the best generalization performance. Besides, TOAST is strongly compatible with existing augmentation-based regularization methods. Our method also obtains a speedup of up to 2.95x compared with a recent state-of-the-art method. |
| 源URL | [http://ir.ia.ac.cn/handle/173211/59452]  |
| 专题 | 自动化研究所_学术期刊_IEEE/CAA Journal of Automatica Sinica |
| 推荐引用方式 GB/T 7714 | Zimo Yin,Jian Pu,Yijie Zhou,et al. Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation[J]. IEEE/CAA Journal of Automatica Sinica,2024,11(11):2270-2283. |
| APA | Zimo Yin,Jian Pu,Yijie Zhou,&Xiangyang Xue.(2024).Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation.IEEE/CAA Journal of Automatica Sinica,11(11),2270-2283. |
| MLA | Zimo Yin,et al."Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation".IEEE/CAA Journal of Automatica Sinica 11.11(2024):2270-2283. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。