Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units
文献类型：期刊论文
| 作者 | Li, Bo1,2,3; Tang, Shanshan4; Yu, Haijun1,2,3 |
| 刊名 | COMMUNICATIONS IN COMPUTATIONAL PHYSICS
 |
| 出版日期 | 2020-02-01 |
| 卷号 | 27期号:2页码:379-411 |
| 关键词 | Deep neural network high dimensional approximation sparse grids rectified linear unit rectified power unit rectified quadratic unit |
| ISSN号 | 1815-2406 |
| DOI | 10.4208/cicp.OA-2019-0168 |
| 英文摘要 | Deep neural networks with rectified linear units (ReLU) are getting more and more popular due to their universal representation power and successful applications. Some theoretical progress regarding the approximation power of deep ReLU network for functions in Sobolev space and Korobov space have recently been made by [D. Yarotsky, Neural Network, 94:103-114, 2017] and [H. Montanelli and Q. Du, SIAM J Math. Data Sci., 1:78-92, 2019], etc. In this paper, we show that deep networks with rectified power units (RePU) can give better approximations for smooth functions than deep ReLU networks. Our analysis bases on classical polynomial approximation theory and some efficient algorithms proposed in this paper to convert polynomials into deep RePU networks of optimal size with no approximation error. Comparing to the results on ReLU networks, the sizes of RePU networks required to approximate functions in Sobolev space and Korobov space with an error tolerance epsilon, by our constructive proofs, are in general O(log1/epsilon) times smaller than the sizes of corresponding ReLU networks constructed in most of the existing literature. Comparing to the classical results of Mhaskar [Mhaskar, Adv. Comput. Math. 1:61-80, 1993], our constructions use less number of activation functions and numerically more stable, they can be served as good initials of deep RePU networks and further trained to break the limit of linear approximation theory. The functions represented by RePU networks are smooth functions, so they naturally fit in the places where derivatives are involved in the loss function. |
| 资助项目 | China National Program on Key Basic Research Project[2015CB856003] ; NNSFC[11771439] ; NNSFC[91852116] ; China Science Challenge Project[TZ2018001] |
| WOS研究方向 | Physics |
| 语种 | 英语 |
| WOS记录号 | WOS:000501534800002 |
| 出版者 | GLOBAL SCIENCE PRESS |
| 源URL | [http://ir.amss.ac.cn/handle/2S8OKBNM/50325]  |
| 专题 | 中国科学院数学与系统科学研究院 |
| 通讯作者 | Yu, Haijun |
| 作者单位 | 1.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China 2.Chinese Acad Sci, Acad Math & Syst Sci, Inst Computat Math & Sci Engn Comp, NCMIS, Beijing 100190, Peoples R China 3.Chinese Acad Sci, Acad Math & Syst Sci, Inst Computat Math & Sci Engn Comp, LSEC, Beijing 100190, Peoples R China 4.China Justice Big Data Inst, Beijing 100043, Peoples R China |
| 推荐引用方式 GB/T 7714 | Li, Bo,Tang, Shanshan,Yu, Haijun. Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units[J]. COMMUNICATIONS IN COMPUTATIONAL PHYSICS,2020,27(2):379-411. |
| APA | Li, Bo,Tang, Shanshan,&Yu, Haijun.(2020).Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units.COMMUNICATIONS IN COMPUTATIONAL PHYSICS,27(2),379-411. |
| MLA | Li, Bo,et al."Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units".COMMUNICATIONS IN COMPUTATIONAL PHYSICS 27.2(2020):379-411. |
入库方式： OAI收割
来源：数学与系统科学研究院
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。