中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 重庆绿色智能技术研究院
  3. 中国科学院重庆绿色智能技术研究院
热门
Activated Gradients for Deep Neural Networks

文献类型：期刊论文

作者Liu, Mei1,2; Chen, Liangming1,2; Du, Xiaohao3; Jin, Long1; Shang, Mingsheng1,2,3
刊名IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
出版日期2021-08-31
页码13
关键词Training Deep learning Neural networks Optimization Visualization Newton method Eigenvalues and eigenfunctions Exploding gradient problems gradient activation function (GAF) ill-conditioned problems saddle point problems vanishing gradient problems
ISSN号2162-237X
DOI10.1109/TNNLS.2021.3106044
通讯作者Jin, Long(longjin@ieee.org)
英文摘要Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this article, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this article gives conditions that the GAF needs to meet and, on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this article proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.
资助项目National Natural Science Foundation of China[62176109] ; National Natural Science Foundation of China[62072429] ; Natural Science Foundation of Chongqing (China)[cstc2020jcyj-zdxmX0028] ; CAS Light of West China Program ; CAAI-Huawei MindSpore Open Fund[CAAIXSJLJJ-2020-009A] ; Gansu Provincial Youth Doctoral Fund of Colleges and Universities[2021QB-003] ; Natural Science Foundation of Gansu Province (China)[20JR10RA639]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:000732099500001
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
源URL[http://119.78.100.138/handle/2HOD01W0/14752]  
专题中国科学院重庆绿色智能技术研究院
通讯作者Jin, Long
作者单位1.Chinese Acad Sci, Chongqing Inst Green & Intelligent Technol, Chongqing Key Lab Big Data & Intelligent Comp, Chongqing 400714, Peoples R China
2.Univ Chinese Acad Sci, Chongqing Sch, Chongqing 400714, Peoples R China
3.Lanzhou Univ, Sch Informat Sci & Engn, Lanzhou 730000, Peoples R China
推荐引用方式
GB/T 7714		 Liu, Mei,Chen, Liangming,Du, Xiaohao,et al. Activated Gradients for Deep Neural Networks[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2021:13.
APA Liu, Mei,Chen, Liangming,Du, Xiaohao,Jin, Long,&Shang, Mingsheng.(2021).Activated Gradients for Deep Neural Networks.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,13.
MLA Liu, Mei,et al."Activated Gradients for Deep Neural Networks".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2021):13.

入库方式： OAI收割

来源：重庆绿色智能技术研究院

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。