CPLDP: an Efficient Large Dataset Processing System Built on Cloud Platform
文献类型：会议论文
| 作者 | Zhiyong Zhong; Mark Li; Jin Chang; Le Zhou; Joshua Zhexue Huang; Shengzhong Feng |
| 出版日期 | 2010 |
| 会议名称 | 6th International Conference on Advanced Data Mining and Applications, ADMA 2010 |
| 英文摘要 | Data intensive applications are widely existed, such as massive data mining, search engine and high-throughput computing in bioinformatics, etc. Data processing becomes a bottleneck as the scale keeps bombing. However, the cost of processing the large scale dataset increases dramatically in traditional relational database, because traditional technology inclines to adopt high performance computer. The boost of cloud computing brings a new solution for data processing due to the characteristics of easy scalability, robustness, large scale storage and high performance. It provides a cost effective platform to implement distributed parallel data processing algorithms. In this paper, we proposed CPLDP (Cloud based Parallel Large Data Processing System), which is an innovative MapReduce based parallel data processing system developed to satisfy the urgent requirements of large data processing. In CPLDP system, we proposed a new method called operation dependency analysis to model data processing workflow and furthermore, reorder and combine some operations when it is possible. Such optimization reduces intermediate file read and write. The performance test proves that the optimization of processing workflow can reduce the time and intermediate results. 漏 2010 Springer-Verlag |
| 收录类别 | EI |
| 语种 | 英语 |
| 源URL | [http://ir.siat.ac.cn:8080/handle/172644/3116]  |
| 专题 | 深圳先进技术研究院_数字所 |
| 作者单位 | 2010 |
| 推荐引用方式 GB/T 7714 | Zhiyong Zhong,Mark Li,Jin Chang,et al. CPLDP: an Efficient Large Dataset Processing System Built on Cloud Platform[C]. 见:6th International Conference on Advanced Data Mining and Applications, ADMA 2010. |
入库方式： OAI收割
来源：深圳先进技术研究院
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。