Stabilize deep ResNet with a sharp scaling factor tau
文献类型：期刊论文
| 作者 | Zhang, Huishuai2; Yu, Da3; Yi, Mingyang4; Chen, Wei1,2; Liu, Tie-Yan |
| 刊名 | MACHINE LEARNING
 |
| 出版日期 | 2022-08-01 |
| 页码 | 34 |
| ISSN号 | 0885-6125 |
| DOI | 10.1007/s10994-022-06192-x |
| 英文摘要 | We study the stability and convergence of training deep ResNets with gradient descent. Specifically, we show that the parametric branch in the residual block should be scaled down by a factor tau=O(1/L) to guarantee stable forward/backward process, where L is the number of residual blocks. Moreover, we establish a converse result that the forward process is unbounded when tau > L-1/2+c for any positive constant c. The above two results together establish a sharp value of the scaling factor in determining the stability of deep ResNet. Based on the stability result, we further show that gradient descent finds the global minima if the ResNet is properly over-parameterized, which significantly improves over the previous work with a much larger range of tau that admits global convergence. Moreover, we show that the convergence rate is independent of the depth, theoretically justifying the advantage of ResNet over vanilla feedforward network. Empirically, with such a factor tau, one can train deep ResNet without normalization layer. Moreover for ResNets with normalization layer, adding such a factor tau also stabilizes the training and obtains significant performance gain for deep ResNet. |
| WOS研究方向 | Computer Science |
| 语种 | 英语 |
| WOS记录号 | WOS:000834697900001 |
| 出版者 | SPRINGER |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/19482]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Zhang, Huishuai; Chen, Wei |
| 作者单位 | 1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Microsoft Res Asia, Beijing, Peoples R China 3.Sun Yat Sen Univ, Guangzhou, Peoples R China 4.Univ Chinese Acad Sci, Beijing, Peoples R China |
| 推荐引用方式 GB/T 7714 | Zhang, Huishuai,Yu, Da,Yi, Mingyang,et al. Stabilize deep ResNet with a sharp scaling factor tau[J]. MACHINE LEARNING,2022:34. |
| APA | Zhang, Huishuai,Yu, Da,Yi, Mingyang,Chen, Wei,&Liu, Tie-Yan.(2022).Stabilize deep ResNet with a sharp scaling factor tau.MACHINE LEARNING,34. |
| MLA | Zhang, Huishuai,et al."Stabilize deep ResNet with a sharp scaling factor tau".MACHINE LEARNING (2022):34. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。