中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search

文献类型：期刊论文

作者Yang, Longxing2,4; Fu, Yanxin1; Lu, Shun2,4; Sun, Zihao2,4; Mei, Jilin4; Zhao, Wenxiao1,3; Hu, Yu2,4
刊名NEURAL NETWORKS
出版日期2023-11-01
卷号168页码:237-255
关键词Zero-shot Neural Architecture Search Low Consistency Sweet Gradient Sweetimator Sweet-enhanced estimators
ISSN号0893-6080
DOI10.1016/j.neunet.2023.09.012
英文摘要Zero-shot Neural Architecture Search has garnered attention due to its training-free nature and rapid search speed. However, existing zero-shot estimators commonly suffer from low consistency, which hampers their practicality. In this work, we theoretically analyze that network generalization and convergence are highly correlated with Sweet Gradient of Parameter, i.e., the number of parameters whose gradient absolute values are within a certain interval. Empirical results indicate that Sweet Gradient of Parameter brings a higher consistency than the overall number of parameters. Additionally, we demonstrate a positive correlation between the network depth and the proportion of parameters with sweet gradients in each layer. Based on the analysis, we propose a training-free method to find the Sweet Gradient interval and obtain an estimator, named Sweetimator. Furthermore, Sweet Gradient can be an effective and general approach to promote the consistency of zero-shot estimators. Experiments show that Sweetimator and Sweet-enhanced estimators have significant consistency improvement in multiple benchmarks. Our method achieves state-of-the-art performance with 256x speedup in NAS-Bench-201 and maintains high competitiveness in DARTS, MobileNet, and Transformer search spaces. The source code is available at https://github.com/xingxing-123/SweetGradient.
资助项目National Natural Science Foundation of China[62176250] ; National Natural Science Foundation of China[62203424] ; National Key Research and Development Program of China[2018YFA0703800] ; Chinese Academy of Sciences (CAS) Project for Young Scientists in Basic Research[YSBR-008] ; Strategic Priority Research Program of Chinese Academy of Sciences[XDA27000000]
WOS研究方向Computer Science ; Neurosciences & Neurology
语种英语
WOS记录号WOS:001086926100001
出版者PERGAMON-ELSEVIER SCIENCE LTD
源URL[http://119.78.100.204/handle/2XEOYT63/21089]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Zhao, Wenxiao; Hu, Yu
作者单位1.Chinese Acad Sci, Acad Math & Syst Sci, Key Lab Syst & Control, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 100049, Peoples R China
3.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China
4.Chinese Acad Sci, Res Ctr Intelligent Comp Syst, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714		 Yang, Longxing,Fu, Yanxin,Lu, Shun,et al. Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search[J]. NEURAL NETWORKS,2023,168:237-255.
APA Yang, Longxing.,Fu, Yanxin.,Lu, Shun.,Sun, Zihao.,Mei, Jilin.,...&Hu, Yu.(2023).Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search.NEURAL NETWORKS,168,237-255.
MLA Yang, Longxing,et al."Sweet Gradient matters: Designing consistent and efficient estimator for Zero-shot Architecture Search".NEURAL NETWORKS 168(2023):237-255.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。