Improving Deep Neural Networks Using Softplus Units
文献类型：会议论文
| 作者 | Hao Zheng1 ; Zhanlei Yang1; Wenju Liu1; Jizhong Liang2; Yanpeng Li2
|
| 出版日期 | 2015 |
| 会议日期 | 2015 |
| 会议地点 | Ireland |
| 关键词 | Softplus Dropout Deep Neural Networks Timit |
| 英文摘要 | Recently, DNNs have achieved great improvement for acoustic modeling in speech recognition tasks. However, it is difficult to train the models well when the depth grows. One main reason is that when training DNNs with traditional sigmoid units, the derivatives damp sharply while back-propagating between layers, which restrict the depth of model especially with insufficient training data. To deal with this problem, some unbounded activation functions have been proposed to preserve sufficient gradients, including ReLU and softplus. Compared with ReLU, the smoothing and nonzero properties of the in gradient makes softplus-based DNNs perform better in both stabilization and performance. However, softplus-based DNNs have been rarely exploited for the phoneme recognition task. In this paper, we explore the use of softplus units for DNNs in acoustic modeling for context-independent phoneme recognition tasks.The revised RBM pre-training and dropout strategy are also applied to improve the performance of softplus units. Experiments show that, the DNNs with softplus units get significantly performance improvement and uses less epochs to get convergence compared to the DNNs trained with standard sigmoid units and ReLUs. |
| 会议录 | IJCNN
 |
| 源URL | [http://ir.ia.ac.cn/handle/173211/11777]  |
| 专题 | 自动化研究所_模式识别国家重点实验室_机器人视觉团队 |
| 通讯作者 | Hao Zheng |
| 作者单位 | 1.National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences 2.Electric Power Research Institute of Shanxi Electric Power Company |
| 推荐引用方式 GB/T 7714 | Hao Zheng,Zhanlei Yang,Wenju Liu,et al. Improving Deep Neural Networks Using Softplus Units[C]. 见:. Ireland. 2015. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。