Hippo: An enhancement of pipeline-aware in-memory caching for HDFS
文献类型：会议论文
| 作者 | Wei, Lan (1) ; Lian, Wenbo (1) ; Liu, Kuien (1) ; Wang, Yongji (1) |
| 出版日期 | 2014 |
| 会议名称 | 2014 23rd International Conference on Computer Communication and Networks, ICCCN 2014 |
| 会议日期 | August 4, 2014 - August 7, 2014 |
| 会议地点 | Shanghai, China |
| 通讯作者 | Wei, Lan |
| 中文摘要 | In the age of big data, distributed computing frameworks tend to coexist and collaborate in pipeline using one scheduler. While a variety of techniques for reducing I/O latency have been proposed, these are rarely specific for the whole pipeline performance. This paper proposes memory management logic called 'Hippo' which targets distributed systems and in particular 'pipelined' applications that might span differing big data frameworks. Though individual frameworks may have internal memory management primitives, Hippo proposes to make a generic framework that works agnostic of these highlevel operations. To increase the hit ratio of in-memory cache, this paper discusses the granularity of caching and how Hippo leverages the job dependency graph to make memory retention and pre-fetching decisions. Our evaluations demonstrate that job dependency is essential to improve the cache performance and a global cache policy maker, in most cases, significantly outperforms explicit caching by users. |
| 英文摘要 | In the age of big data, distributed computing frameworks tend to coexist and collaborate in pipeline using one scheduler. While a variety of techniques for reducing I/O latency have been proposed, these are rarely specific for the whole pipeline performance. This paper proposes memory management logic called 'Hippo' which targets distributed systems and in particular 'pipelined' applications that might span differing big data frameworks. Though individual frameworks may have internal memory management primitives, Hippo proposes to make a generic framework that works agnostic of these highlevel operations. To increase the hit ratio of in-memory cache, this paper discusses the granularity of caching and how Hippo leverages the job dependency graph to make memory retention and pre-fetching decisions. Our evaluations demonstrate that job dependency is essential to improve the cache performance and a global cache policy maker, in most cases, significantly outperforms explicit caching by users. |
| 收录类别 | EI |
| 会议录出版地 | Institute of Electrical and Electronics Engineers Inc. |
| 语种 | 英语 |
| ISSN号 | 10952055 |
| ISBN号 | 9781479935727 |
| 源URL | [http://ir.iscas.ac.cn/handle/311060/16610]  |
| 专题 | 软件研究所_软件所图书馆_会议论文 |
| 推荐引用方式 GB/T 7714 | Wei, Lan ,Lian, Wenbo ,Liu, Kuien ,et al. Hippo: An enhancement of pipeline-aware in-memory caching for HDFS[C]. 见:2014 23rd International Conference on Computer Communication and Networks, ICCCN 2014. Shanghai, China. August 4, 2014 - August 7, 2014. |
入库方式： OAI收割
来源：软件研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。