Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition
文献类型：期刊论文
| 作者 | Yang, Chuanguang2,5; An, Zhulin5; Zhou, Helong3; Zhuang, Fuzhen1,4; Xu, Yongjun5; Zhang, Qian |
| 刊名 | IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
 |
| 出版日期 | 2023-08-01 |
| 卷号 | 45期号:8页码:10212-10227 |
| ISSN号 | 0162-8828 |
| 关键词 | Contrastive learning mutual learning online knowledge distillation visual recognition |
| DOI | 10.1109/TPAMI.2023.3257878 |
| 英文摘要 | The teacher-free online Knowledge Distillation (KD) aims to train an ensemble of multiple student models collaboratively and distill knowledge from each other. Although existing online KD methods achieve desirable performance, they often focus on class probabilities as the core knowledge type, ignoring the valuable feature representational information. We present a Mutual Contrastive Learning (MCL) framework for online KD. The core idea ofMCLis to perform mutual interaction and transfer of contrastive distributions among a cohort of networks in an online manner. Our MCL can aggregate cross-network embedding information and maximize the lower bound to the mutual information between two networks. This enables each network to learn extra contrastive knowledge from others, leading to better feature representations, thus improving the performance of visual recognition tasks. Beyond the final layer, we extend MCL to intermediate layers and perform an adaptive layer-matching mechanism trained by meta-optimization. Experiments on image classification and transfer learning to visual recognition tasks show that layer-wise MCL can lead to consistent performance gains against state-of-the-art online KD approaches. The superiority demonstrates that layer-wise MCL can guide the network to generate better feature representations. Our code is publicly avaliable at https://github.com/winycg/L-MCL. |
| 资助项目 | National Key Research and Development Program of China[2021ZD0113602] ; NationalNatural Science Foundation ofChina[62176014] ; Fundamental Research Funds for the CentralUniversities |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| 出版者 | IEEE COMPUTER SOC |
| WOS记录号 | WOS:001022958600062 |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/21333]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | An, Zhulin |
| 作者单位 | 1.Beihang Univ, Inst Artificial Intelligence, Beijing 100191, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Horizon Robot, Beijing 100089, Peoples R China 4.Zhongguancun Lab, Beijing 100194, Peoples R China 5.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Yang, Chuanguang,An, Zhulin,Zhou, Helong,et al. Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(8):10212-10227. |
| APA | Yang, Chuanguang,An, Zhulin,Zhou, Helong,Zhuang, Fuzhen,Xu, Yongjun,&Zhang, Qian.(2023).Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(8),10212-10227. |
| MLA | Yang, Chuanguang,et al."Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.8(2023):10212-10227. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。