中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 多模态人工智能系统全国重点实验室
  5. 先进控制与自动化团队
Drill the Cork of Information Bottleneck by Inputting the Most Important Data

文献类型：期刊论文

作者Peng, Xinyu3; Zhang, Jiawei3; Wang, Fei-Yue2; Li, Li1
刊名IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
出版日期2021-05-23
页码13
关键词Training Signal to noise ratio Mutual information Optimization Convergence Deep learning Tools Information bottleneck (IB) theory machine learning minibatch stochastic gradient descent (SGD) typicality sampling
ISSN号2162-237X
DOI10.1109/TNNLS.2021.3079112
通讯作者Li, Li(li-li@tsinghua.edu.cn)
英文摘要Deep learning has become the most powerful machine learning tool in the last decade. However, how to efficiently train deep neural networks remains to be thoroughly solved. The widely used minibatch stochastic gradient descent (SGD) still needs to be accelerated. As a promising tool to better understand the learning dynamic of minibatch SGD, the information bottleneck (IB) theory claims that the optimization process consists of an initial fitting phase and the following compression phase. Based on this principle, we further study typicality sampling, an efficient data selection method, and propose a new explanation of how it helps accelerate the training process of the deep networks. We show that the fitting phase depicted in the IB theory will be boosted with a high signal-to-noise ratio of gradient approximation if the typicality sampling is appropriately adopted. Furthermore, this finding also implies that the prior information of the training set is critical to the optimization process, and the better use of the most important data can help the information flow through the bottleneck faster. Both theoretical analysis and experimental results on synthetic and real-world datasets demonstrate our conclusions.
WOS关键词DEEP NEURAL-NETWORKS
资助项目National Key Research and Development Program of China[2018AAA0101402] ; National Natural Science Foundation of China[U1811463]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:000732265100001
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
资助机构National Key Research and Development Program of China ; National Natural Science Foundation of China
源URL[http://ir.ia.ac.cn/handle/173211/46812]  
专题自动化研究所_复杂系统管理与控制国家重点实验室_先进控制与自动化团队
通讯作者Li, Li
作者单位1.Tsinghua Univ, BNRist, Dept Automat, Beijing 100084, Peoples R China
2.Chinese Acad Sci, State Key Lab Management & Control Complex Syst, Inst Automat, Beijing 100080, Peoples R China
3.Tsinghua Univ, Dept Automat, Beijing 100084, Peoples R China
推荐引用方式
GB/T 7714		 Peng, Xinyu,Zhang, Jiawei,Wang, Fei-Yue,et al. Drill the Cork of Information Bottleneck by Inputting the Most Important Data[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2021:13.
APA Peng, Xinyu,Zhang, Jiawei,Wang, Fei-Yue,&Li, Li.(2021).Drill the Cork of Information Bottleneck by Inputting the Most Important Data.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,13.
MLA Peng, Xinyu,et al."Drill the Cork of Information Bottleneck by Inputting the Most Important Data".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2021):13.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。