ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG
文献类型：期刊论文
| 作者 | Qi, Jiaxing1; Xiao, Wencong3; Li, Mingzhen2; Yang, Chaojie4; Li, Yong4; Lin, Wei3; Yang, Hailong1; Luan, Zhongzhi1; Qian, Depei1 |
| 刊名 | IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
 |
| 出版日期 | 2024-10-01 |
| 卷号 | 35期号:10页码:1708-1720 |
| 关键词 | Graphics processing units Dynamic scheduling Throughput Processor scheduling Pipelines Costs Quality of service MIG batch inference scheduling system machine learning |
| ISSN号 | 1045-9219 |
| DOI | 10.1109/TPDS.2024.3431189 |
| 英文摘要 | As deep learning (DL) technologies become ubiquitous, GPU clusters are deployed for inference tasks with consistent service level objectives (SLOs). Efficiently utilizing multiple GPUs is crucial for throughput and cost-effectiveness. This article addresses the challenges posed by dynamic input and NVIDIA MIG in scheduling DL workloads. We present ElasticBatch, a scheduling system that simplifies configuration through bucketization and employs a machine learning-based pipeline to optimize settings. Our experiments demonstrate that ElasticBatch achieves a 50% reduction in GPU instances compared to MIG disablement, increases GPU utilization by 1.4% to 6.5% over an ideal scheduler and significantly reduces profiling time. This research contributes to the discourse on efficient utilization of GPU clusters. ElasticBatch's effectiveness in mitigating challenges posed by dynamic inputs and NVIDIA MIG underscores its potential to optimize GPU cluster performance, providing tangible benefits in terms of reduced instances, increased utilization, and significant time savings in real-world deployment scenarios. |
| 资助项目 | National Key RD Program[2023YFB3001903] ; National Natural Science Foundation of China[62322201] ; National Natural Science Foundation of China[62072018] ; National Natural Science Foundation of China[U23B2020] ; National Natural Science Foundation of China[U22A2028] ; Academic Excellence Foundation of BUAA for PhD Students ; China National Postdoctoral Program for Innovative Talents[BX20240383] |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:001316110600001 |
| 出版者 | IEEE COMPUTER SOC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/39583]  |
| 专题 | 中国科学院计算技术研究所期刊论文_英文 |
| 通讯作者 | Luan, Zhongzhi |
| 作者单位 | 1.Beihang Univ, Sino German Joint Software Inst, Beijing 100191, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China 3.Alibaba Grp, Hangzhou 310052, Zhejiang, Peoples R China 4.Alibaba Grp, Beijing 100102, Peoples R China |
| 推荐引用方式 GB/T 7714 | Qi, Jiaxing,Xiao, Wencong,Li, Mingzhen,et al. ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2024,35(10):1708-1720. |
| APA | Qi, Jiaxing.,Xiao, Wencong.,Li, Mingzhen.,Yang, Chaojie.,Li, Yong.,...&Qian, Depei.(2024).ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,35(10),1708-1720. |
| MLA | Qi, Jiaxing,et al."ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 35.10(2024):1708-1720. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。