中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 多模态人工智能系统全国重点实验室
  5. 人工智能与机器学习
Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints

文献类型：期刊论文

作者Tao, Wei3; Wu, Gao-Wei1,2; Tao, Qing4
刊名IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
出版日期2022-03-01
卷号33期号:3页码:1107-1118
关键词Heavy-ball (HB) methods individual convergence machine learning momentum methods nonsmooth optimization sparsity
ISSN号2162-237X
DOI10.1109/TNNLS.2020.3040325
通讯作者Tao, Qing(taoqing@gmail.com)
英文摘要Momentum technique has recently emerged as an effective strategy in accelerating convergence of gradient descent (GD) methods and exhibits improved performance in deep learning as well as regularized learning. Typical momentum examples include Nesterov's accelerated gradient (NAG) and heavy-ball (HB) methods. However, so far, almost all the acceleration analyses are only limited to NAG, and a few investigations about the acceleration of HB are reported. In this article, we address the convergence about the last iterate of HB in nonsmooth optimizations with constraints, which we name individual convergence. This question is significant in machine learning, where the constraints are required to impose on the learning structure and the individual output is needed to effectively guarantee this structure while keeping an optimal rate of convergence. Specifically, we prove that HB achieves an individual convergence rate of O(1/root t), where t is the number of iterations. This indicates that both of the two momentum methods can accelerate the individual convergence of basic GD to be optimal. Even for the convergence of averaged iterates, our result avoids the disadvantages of the previous work in restricting the optimization problem to be unconstrained as well as limiting the performed number of iterations to be predefined. The novelty of convergence analysis presented in this article provides a clear understanding of how the HB momentum can accelerate the individual convergence and reveals more insights about the similarities and differences in getting the averaging and individual convergence rates. The derived optimal individual convergence is extended to regularized and stochastic settings, in which an individual solution can be produced by the projection-based operation. In contrast to the averaged output, the sparsity can be reduced remarkably without sacrificing the theoretical optimal rates. Several real experiments demonstrate the performance of HB momentum strategy.
资助项目NSFC[62076252] ; NSFC[61673394] ; NSFC[61976213] ; Beijing Advanced Discipline Fund
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:000766269100021
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
资助机构NSFC ; Beijing Advanced Discipline Fund
源URL[http://ir.ia.ac.cn/handle/173211/48166]  
专题精密感知与控制研究中心_人工智能与机器学习
通讯作者Tao, Qing
作者单位1.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China
2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
3.Army Engn Univ PLA, Coll Command & Control Engn, Nanjing 210007, Peoples R China
4.Army Acad Artillery & Air Def, Hefei 230031, Peoples R China
推荐引用方式
GB/T 7714		 Tao, Wei,Wu, Gao-Wei,Tao, Qing. Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2022,33(3):1107-1118.
APA Tao, Wei,Wu, Gao-Wei,&Tao, Qing.(2022).Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,33(3),1107-1118.
MLA Tao, Wei,et al."Momentum Acceleration in the Individual Convergence of Nonsmooth Convex Optimization With Constraints".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 33.3(2022):1107-1118.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。