Probing Language Models from A Human Behavioral Perspective
文献类型:期刊论文
作者 | Wang, Xintong3; Li, Xiaoyu2; Li, Xingshan1; Biemann, Chris3 |
刊名 | arXiv |
出版日期 | 2023 |
DOI | 10.48550/arXiv.2310.05216 |
文献子类 | 综述 |
英文摘要 | Large Language Models (LLMs) have emerged as dominant foundational models in modern NLP. However, the understanding of their prediction process and internal mechanisms, such as feed-forward networks and multi-head self-attention, remains largely unexplored. In this study, we probe LLMs from a human behavioral perspective, correlating values from LLMs with eye-tracking measures, which are widely recognized as meaningful indicators of reading patterns. Our findings reveal that LLMs exhibit a prediction pattern distinct from that of RNN-based LMs. Moreover, with the escalation of FFN layers, the capacity for memorization and linguistic knowledge encoding also surges until it peaks, subsequently pivoting to focus on comprehension capacity. The functions of self-attention are distributed across multiple heads. Lastly, we scrutinize the gate mechanisms, finding that they control the flow of information, with some gates promoting, while others eliminating information. |
收录类别 | EI |
语种 | 英语 |
源URL | [http://ir.psych.ac.cn/handle/311026/46208] |
专题 | 中国科学院心理研究所 |
作者单位 | 1.Department of Informatics, Technische Universität Berlin, Germany 2.Institute of Psychology, Chinese Academy of Sciences, China 3.Department of Informatics, Universität Hamburg, Germany |
推荐引用方式 GB/T 7714 | Wang, Xintong,Li, Xiaoyu,Li, Xingshan,et al. Probing Language Models from A Human Behavioral Perspective[J]. arXiv,2023. |
APA | Wang, Xintong,Li, Xiaoyu,Li, Xingshan,&Biemann, Chris.(2023).Probing Language Models from A Human Behavioral Perspective.arXiv. |
MLA | Wang, Xintong,et al."Probing Language Models from A Human Behavioral Perspective".arXiv (2023). |
入库方式: OAI收割
来源:心理研究所
其他版本
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。