中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Workload-Adaptive Configuration Tuning for Hierarchical Cloud Schedulers

文献类型：期刊论文

作者Han, Rui1; Liu, Chi Harold1; Zong, Zan2; Chen, Lydia Y.3; Liu, Wending1; Wang, Siyi4; Zhan, Jianfeng4
刊名IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
出版日期2019-12-01
卷号30期号:12页码:2879-2895
关键词Resource management Scheduling Google Yarn Tuning Facebook Task analysis Cloud datacenter cluster scheduler configuration job latency YARN
ISSN号1045-9219
DOI10.1109/TPDS.2019.2923197
英文摘要Cluster schedulers provide flexible resource sharing mechanism for best-effort cloud jobs, which occupy a majority in modern datacenters. Properly tuning a scheduler & x0027;s configurations is the key to these jobs' performance because it decides how to allocate resources among them. Today & x0027;s cloud scheduling systems usually rely on cluster operators to set the configuration and thus overlook the potential performance improvement through optimally configuring the scheduler according to the heterogeneous and dynamic cloud workloads. In this paper, we introduce AdaptiveConfig, a run-time configurator for cluster schedulers that automatically adapts to the changing workload and resource status in two steps. First, a comparison approach estimates jobs' performances under different configurations and diverse scheduling scenarios. The key idea here is to transform a scheduler & x0027;s resource allocation mechanism and their variable influence factors (configurations, scheduling constraints, available resources, and workload status) into business rules and facts in a rule engine, thereby reasoning about these correlated factors in job performance comparison. Second, a workload-adaptive optimizer transforms the cluster-level searching of huge configuration space into an equivalent dynamic programming problem that can be efficiently solved at scale. We implement AdaptiveConfig on the popular YARN Capacity and Fair schedulers and demonstrate its effectiveness using real-world Facebook and Google workloads, i.e., successfully finding best configurations for most of scheduling scenarios and considerably reducing latencies by a factor of two with low optimization time.
资助项目National Key Research and Development Plan of China[2018YFB1003701] ; National Key Research and Development Plan of China[2018YFB1003700] ; National Natural Science Foundation of China[61872337]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:000498569400019
出版者IEEE COMPUTER SOC
源URL[http://119.78.100.204/handle/2XEOYT63/14956]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Liu, Chi Harold
作者单位1.Beijing Inst Technol, Beijing Shi 100091, Peoples R China
2.Tsinghua Univ, Beijing Shi 100091, Peoples R China
3.Delft Univ Technol, NL-2628 CD Delft, Netherlands
4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式
GB/T 7714		 Han, Rui,Liu, Chi Harold,Zong, Zan,et al. Workload-Adaptive Configuration Tuning for Hierarchical Cloud Schedulers[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2019,30(12):2879-2895.
APA Han, Rui.,Liu, Chi Harold.,Zong, Zan.,Chen, Lydia Y..,Liu, Wending.,...&Zhan, Jianfeng.(2019).Workload-Adaptive Configuration Tuning for Hierarchical Cloud Schedulers.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,30(12),2879-2895.
MLA Han, Rui,et al."Workload-Adaptive Configuration Tuning for Hierarchical Cloud Schedulers".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 30.12(2019):2879-2895.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。