A new data access mechanism for hdfs
文献类型：期刊论文
| 作者 | Li,Qiang1,2; Sun,Zhenyu1,2; Wei,Zhanchen1,2; Sun,Gongxing1 |
| 刊名 | Journal of physics: conference series
 |
| 出版日期 | 2017-10-01 |
| 卷号 | 898期号:6 |
| ISSN号 | 1742-6588 |
| DOI | 10.1088/1742-6596/898/6/062018 |
| 英文摘要 | Abstract with the era of big data emerging, hadoop has become the de facto standard of big data processing platform. however, it is still difficult to get legacy applications, such as high energy physics (hep) applications, to run efficiently on hadoop platform. there are two reasons which lead to the difficulties mentioned above: firstly, random access is not supported on hadoop file system (hdfs), secondly, it is difficult to make legacy applications adopt to hdfs streaming data processing mode. in order to address the two issues, a new read and write mechanism of hdfs is proposed. with this mechanism, data access is done on the local file system instead of through hdfs streaming interfaces. to enable files modified by users, three attributes including permissions, owner and group are imposed on block objects. blocks stored on datanodes have the same attributes as the file they are owned by. users can modify blocks when the map task running locally, and hdfs is responsible to update the rest replicas later after the block modification finished. to further improve the performance of hadoop system, a complete localization task execution mechanism is implemented for i/o intensive jobs. test results show that average cpu utilization is improved by 10% with the new task selection strategy, data read and write performances are improved by about 10% and 30% separately. |
| 语种 | 英语 |
| WOS记录号 | IOP:1742-6588-898-6-062018 |
| 出版者 | IOP Publishing |
| URI标识 | http://www.irgrid.ac.cn/handle/1471x/2175688 |
| 专题 | 高能物理研究所 |
| 作者单位 | 1.Institute of High Energy Physics, Beijing, China 2.University of Chinese Academy of Sciences, Beijing, China |
| 推荐引用方式 GB/T 7714 | Li,Qiang,Sun,Zhenyu,Wei,Zhanchen,et al. A new data access mechanism for hdfs[J]. Journal of physics: conference series,2017,898(6). |
| APA | Li,Qiang,Sun,Zhenyu,Wei,Zhanchen,&Sun,Gongxing.(2017).A new data access mechanism for hdfs.Journal of physics: conference series,898(6). |
| MLA | Li,Qiang,et al."A new data access mechanism for hdfs".Journal of physics: conference series 898.6(2017). |
入库方式： iSwitch采集
来源：高能物理研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。