Probabilistic Automata-Based Method for Enhancing Performance of Deep Reinforcement Learning Systems
文献类型：期刊论文
| 作者 | Min Yang; Guanjun Liu; Ziyuan Zhou; Jiacun Wang |
| 刊名 | IEEE/CAA Journal of Automatica Sinica
 |
| 出版日期 | 2024 |
| 卷号 | 11期号:11页码:2327-2339 |
| 关键词 | Deep reinforcement learning (DRL) performance improvement framework probabilistic automata real-time monitoring the key probabilistic decision-making units (PDMU)-action pair |
| ISSN号 | 2329-9266 |
| DOI | 10.1109/JAS.2024.124818 |
| 英文摘要 | Deep reinforcement learning (DRL) has demonstrated significant potential in industrial manufacturing domains such as workshop scheduling and energy system management. However, due to the model’s inherent uncertainty, rigorous validation is requisite for its application in real-world tasks. Specific tests may reveal inadequacies in the performance of pre-trained DRL models, while the “black-box” nature of DRL poses a challenge for testing model behavior. We propose a novel performance improvement framework based on probabilistic automata, which aims to proactively identify and correct critical vulnerabilities of DRL systems, so that the performance of DRL models in real tasks can be improved with minimal model modifications. First, a probabilistic automaton is constructed from the historical trajectory of the DRL system by abstracting the state to generate probabilistic decision-making units (PDMUs), and a reverse breadth-first search (BFS) method is used to identify the key PDMU-action pairs that have the greatest impact on adverse outcomes. This process relies only on the state-action sequence and final result of each trajectory. Then, under the key PDMU, we search for the new action that has the greatest impact on favorable results. Finally, the key PDMU, undesirable action and new action are encapsulated as monitors to guide the DRL system to obtain more favorable results through real-time monitoring and correction mechanisms. Evaluations in two standard reinforcement learning environments and three actual job scheduling scenarios confirmed the effectiveness of the method, providing certain guarantees for the deployment of DRL models in real-world applications. |
| 源URL | [http://ir.ia.ac.cn/handle/173211/59456]  |
| 专题 | 自动化研究所_学术期刊_IEEE/CAA Journal of Automatica Sinica |
| 推荐引用方式 GB/T 7714 | Min Yang,Guanjun Liu,Ziyuan Zhou,et al. Probabilistic Automata-Based Method for Enhancing Performance of Deep Reinforcement Learning Systems[J]. IEEE/CAA Journal of Automatica Sinica,2024,11(11):2327-2339. |
| APA | Min Yang,Guanjun Liu,Ziyuan Zhou,&Jiacun Wang.(2024).Probabilistic Automata-Based Method for Enhancing Performance of Deep Reinforcement Learning Systems.IEEE/CAA Journal of Automatica Sinica,11(11),2327-2339. |
| MLA | Min Yang,et al."Probabilistic Automata-Based Method for Enhancing Performance of Deep Reinforcement Learning Systems".IEEE/CAA Journal of Automatica Sinica 11.11(2024):2327-2339. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。