Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
文献类型:会议论文
作者 | Peng HW(彭厚文)4; Du, Hao3; Yu HY(俞宏远)2; Li, Qi1; Liao, Jing3; Fu, Jianlong4 |
出版日期 | 2020 |
会议日期 | 2020 |
会议地点 | Vancouver, Canada |
英文摘要 | One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. However, weight sharing across models has an inherent deficiency, i.e., insufficient training of subnetworks in hypernetworks. To alleviate this problem, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop. We directly select the most promising one from the prioritized paths as the final architecture, without using other complex search methods, such as reinforcement learning or evolution algorithms. The experiments on ImageNet verify such path distillation method can improve the convergence ratio and performance of the hypernetwork, as well as boosting the training of subnetworks. The discovered architectures achieve superior performance compared to the recent MobileNetV3 and EfficientNet families under aligned settings. Moreover, the experiments on object detection and more challenging search space show the generality and robustness of the proposed method. Code and models are available at https://github.com/microsoft/cream.git. |
会议录出版者 | NeurIPS |
会议录出版地 | NeurIPS |
源URL | [http://ir.ia.ac.cn/handle/173211/48710] |
专题 | 自动化研究所_智能感知与计算研究中心 |
通讯作者 | Peng HW(彭厚文) |
作者单位 | 1.Tsinghua University 2.Chinese Academy of Sciences 3.City University of Hong Kong 4.Microsoft Research Asia |
推荐引用方式 GB/T 7714 | Peng HW,Du, Hao,Yu HY,et al. Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search[C]. 见:. Vancouver, Canada. 2020. |
入库方式: OAI收割
来源:自动化研究所
其他版本
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。