MOSS: An Open Conversational Large Language Model
文献类型：期刊论文
| 作者 | Tianxiang Sun; Xiaotian Zhang; Zhengfu He; Peng Li ; Qinyuan Cheng; Xiangyang Liu; Hang Yan; Yunfan Shao; Qiong Tang; Shiduo Zhang
|
| 刊名 | Machine Intelligence Research
 |
| 出版日期 | 2024 |
| 卷号 | 21期号:5页码:888-905 |
| 关键词 | Large language models natural language processing pre-training alignment chatGPT MOSS |
| ISSN号 | 2731-538X |
| DOI | 10.1007/s11633-024-1502-8 |
| 英文摘要 | Conversational large language models (LLMs) such as ChatGPT and GPT-4 have recently exhibited remarkable capabilities across various domains, capturing widespread attention from the public. To facilitate this line of research, in this paper, we report the development of MOSS, an open-sourced conversational LLM that contains 16B parameters and can perform a variety of instructions in multi-turn interactions with humans. The base model of MOSS is pre-trained on large-scale unlabeled English, Chinese, and code data. To optimize the model for dialogue, we generate 1.1M synthetic conversations based on user prompts collected through our earlier versions of the model API. We then perform preference-aware training on preference data annotated from AI feedback. Evaluation results on real-world use cases and academic benchmarks demonstrate the effectiveness of the proposed approaches. In addition, we present an effective practice to augment MOSS with several external tools. Through the development of MOSS, we have established a complete technical roadmap for large language models from pre-training, supervised fine-tuning to alignment, verifying the feasibility of chatGPT under resource-limited conditions and providing a reference for both the academic and industrial communities. Model weights and code are publicly available at https://github.com/OpenMOSS/MOSS. |
| 源URL | [http://ir.ia.ac.cn/handle/173211/59420]  |
| 专题 | 自动化研究所_学术期刊_International Journal of Automation and Computing |
| 作者单位 | Fudan University, Shanghai 200438, China |
| 推荐引用方式 GB/T 7714 | Tianxiang Sun,Xiaotian Zhang,Zhengfu He,et al. MOSS: An Open Conversational Large Language Model[J]. Machine Intelligence Research,2024,21(5):888-905. |
| APA | Tianxiang Sun.,Xiaotian Zhang.,Zhengfu He.,Peng Li.,Qinyuan Cheng.,...&Xipeng Qiu.(2024).MOSS: An Open Conversational Large Language Model.Machine Intelligence Research,21(5),888-905. |
| MLA | Tianxiang Sun,et al."MOSS: An Open Conversational Large Language Model".Machine Intelligence Research 21.5(2024):888-905. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。