Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding
文献类型：期刊论文
| 作者 | Zhu, Yunchang1,2; Pang, Liang1; Wu, Kangxi1,2; Lan, Yanyan3; Shen, Huawei1,2; Cheng, Xueqi1,2 |
| 刊名 | ACM TRANSACTIONS ON INFORMATION SYSTEMS
 |
| 出版日期 | 2024-09-01 |
| 卷号 | 42期号:5页码:29 |
| 关键词 | Natural language understanding question answering pseudo-relevance feedback loss function |
| ISSN号 | 1046-8188 |
| DOI | 10.1145/3652599 |
| 英文摘要 | Current natural language understanding (NLU) models have been continuously scaling up, both in terms of model size and input context, introducing more hidden and input neurons. While this generally improves performance on average, the extra neurons do not yield a consistent improvement for all instances. This is because some hidden neurons are redundant, and the noise mixed in input neurons tends to distract the model. Previous work mainly focuses on extrinsically reducing low-utility neurons by additional post- or pre-processing, such as network pruning and context selection, to avoid this problem. Beyond that, can we make the model reduce redundant parameters and suppress input noise by intrinsically enhancing the utility of each neuron? If a model can efficiently utilize neurons, no matter which neurons are ablated (disabled), the ablated submodel should perform no better than the original full model. Based on such a comparison principle between models, we propose a cross-model comparative loss for a broad range of tasks. Comparative loss is essentially a ranking loss on top of the task-specific losses of the full and ablated models, with the expectation that the task-specific loss of the full model isminimal. We demonstrate the universal effectiveness of comparative loss through extensive experiments on 14 datasets fromthree distinctNLU tasks based on fivewidely used pre-trained language models and find it particularly superior for models with few parameters or long input. |
| 资助项目 | National Key R&D Program of China[2022YFB3103700] ; National Key R&D Program of China[2022YFB3103704] ; National Natural Science Foundation of China (NSFC)[62276248] ; National Natural Science Foundation of China (NSFC)[U21B2046] ; Youth Innovation Promotion Association CAS[2023111] |
| WOS研究方向 | Computer Science |
| 语种 | 英语 |
| WOS记录号 | WOS:001253867000011 |
| 出版者 | ASSOC COMPUTING MACHINERY |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/39842]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Pang, Liang |
| 作者单位 | 1.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab AI Secur, 6 Kexueyuan South Rd, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, 6 Kexueyuan South Rd, Beijing 100190, Peoples R China 3.Tsinghua Univ, Inst AI Ind Res, 30 Shuangqing Rd, Beijing 100084, Peoples R China |
| 推荐引用方式 GB/T 7714 | Zhu, Yunchang,Pang, Liang,Wu, Kangxi,et al. Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding[J]. ACM TRANSACTIONS ON INFORMATION SYSTEMS,2024,42(5):29. |
| APA | Zhu, Yunchang,Pang, Liang,Wu, Kangxi,Lan, Yanyan,Shen, Huawei,&Cheng, Xueqi.(2024).Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding.ACM TRANSACTIONS ON INFORMATION SYSTEMS,42(5),29. |
| MLA | Zhu, Yunchang,et al."Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding".ACM TRANSACTIONS ON INFORMATION SYSTEMS 42.5(2024):29. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。