Automating Characterization Deployment in Distributed Data Stream Management Systems
文献类型：期刊论文
| 作者 | Wang, Chunkai1; Meng, Xiaofeng1; Guo, Qi2; Weng, Zujian1; Yang, Chen1 |
| 刊名 | IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
 |
| 出版日期 | 2017-12-01 |
| 卷号 | 29期号:12页码:2669-2681 |
| 关键词 | Stream processing system relational query system incremental learning modeling and prediction |
| ISSN号 | 1041-4347 |
| DOI | 10.1109/TKDE.2017.2751606 |
| 英文摘要 | Distributed data stream management systems (DDSMS) are usually composed of upper layer relational query systems (RQS) and lower layer stream processing systems (SPS). When users submit new queries to RQS, a query planner needs to be converted into a directed acyclic graph (DAG) consisting of tasks which are running on SPS. Based on different query requests and data stream properties, SPS need to configure different deployments strategies. However, how to dynamically predict deployment configurations of SPS to ensure the processing throughput and low resource usage is a great challenge. This article presents OrientStream, a framework for automating characterization deployment in DDSMS using incremental machine learning techniques. By introducing the data-level, query plan-level, operator-level, and cluster-level's four-level feature extraction mechanism, we first use the different query workloads as training sets to predict the resource usage by DDSMS, and select the optimal resource configuration from candidate settings based on the current query requests and stream properties, then migrate the operator state by introducing dynamic reconfiguration. Finally, we validate our approach on the open source SPS-Storm. In view of the application scenarios with long monitoring cycle and non-frequent data fluctuation, experiments show that OrientStream can reduce CPU usage of 8-15 percent and memory usage of 38-48 percent, respectively. |
| 资助项目 | Natural Science Foundation of China[91646203] ; Natural Science Foundation of China[61379050] ; Natural Science Foundation of China[61532016] ; Natural Science Foundation of China[61532010] ; Natural Science Foundation of China[61762082] ; National Key Research and Development Program of China[2016YFB1000602] ; National Key Research and Development Program of China[2016YFB1000603] ; Renmin University[11XNL010] ; Science and Technology Opening up Cooperation project of Henan Province[172106000077] |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:000414712700004 |
| 出版者 | IEEE COMPUTER SOC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/6490]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Meng, Xiaofeng |
| 作者单位 | 1.Renmin Univ China, Sch Informat, Beijing 100872, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Wang, Chunkai,Meng, Xiaofeng,Guo, Qi,et al. Automating Characterization Deployment in Distributed Data Stream Management Systems[J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,2017,29(12):2669-2681. |
| APA | Wang, Chunkai,Meng, Xiaofeng,Guo, Qi,Weng, Zujian,&Yang, Chen.(2017).Automating Characterization Deployment in Distributed Data Stream Management Systems.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,29(12),2669-2681. |
| MLA | Wang, Chunkai,et al."Automating Characterization Deployment in Distributed Data Stream Management Systems".IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 29.12(2017):2669-2681. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。