A New Data Access Mechanism for HDFS
文献类型：期刊论文
| 作者 | Wei, Zhanchen; Sun, Zhenyu ; Li, Qiang; Sun GX(孙功星); Wei ZC(魏占辰); Sun, Gongxing; Sun ZY(孙震宇); Li Q(李强)
|
| 刊名 | Journal of Physics: Conference Series
 |
| 出版日期 | 2017 |
| 卷号 | 898期号:6页码:062018 |
| ISSN号 | 1742-6588 |
| DOI | 10.1088/1742-6596/898/6/062018 |
| 文献子类 | Proceedings Paper |
| 英文摘要 | With the era of big data emerging, Hadoop has become the de facto standard of big data processing platform. However, it is still difficult to get legacy applications, such as High Energy Physics (HEP) applications, to run efficiently on Hadoop platform. There are two reasons which lead to the difficulties mentioned above: firstly, random access is not supported on Hadoop File System (HDFS), secondly, it is difficult to make legacy applications adopt to HDFS streaming data processing mode. In order to address the two issues, a new read and write mechanism of HDFS is proposed. With this mechanism, data access is done on the local file system instead of through HDFS streaming interfaces. To enable files modified by users, three attributes including permissions, owner and group are imposed on Block objects. Blocks stored on Datanodes have the same attributes as the file they are owned by. Users can modify blocks when the Map task running locally, and HDFS is responsible to update the rest replicas later after the block modification finished. To further improve the performance of Hadoop system, a complete localization task execution mechanism is implemented for I/O intensive jobs. Test results show that average CPU utilization is improved by 10% with the new task selection strategy, data read and write performances are improved by about 10% and 30% separately. © Published under licence by IOP Publishing Ltd. |
| 会议地点 | San Francisco, CA, United states |
| 电子版国际标准刊号 | 1742-6596 |
| 会议日期 | October 10, 2016 - October 14, 2016 |
| 语种 | 英语 |
| 源URL | [http://ir.ihep.ac.cn/handle/311005/285436]  |
| 专题 | 中国科学院高能物理研究所 |
| 作者单位 | 1.University of Chinese, Academy of Sciences, Beijing, China 2.Institute of High Energy Physics, Beijing, China; |
| 推荐引用方式 GB/T 7714 | Wei, Zhanchen,Sun, Zhenyu,Li, Qiang,et al. A New Data Access Mechanism for HDFS[J]. Journal of Physics: Conference Series,2017,898(6):062018. |
| APA | Wei, Zhanchen.,Sun, Zhenyu.,Li, Qiang.,孙功星.,魏占辰.,...&李强.(2017).A New Data Access Mechanism for HDFS.Journal of Physics: Conference Series,898(6),062018. |
| MLA | Wei, Zhanchen,et al."A New Data Access Mechanism for HDFS".Journal of Physics: Conference Series 898.6(2017):062018. |
入库方式： OAI收割
来源：高能物理研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。