中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 计算技术研究所
  3. 中国科学院计算技术研究所
  4. 中国科学院计算技术研究所期刊论文
  5. 英文
Soft Error Reliability Analysis of Vision Transformers

文献类型：期刊论文

作者Xue, Xinghua1,2; Liu, Cheng1,2; Wang, Ying1,2; Yang, Bing3; Luo, Tao4; Zhang, Lei1,2; Li, Huawei1,2; Li, Xiaowei1,2
刊名IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS
出版日期2023-10-05
页码11
ISSN号1063-8210
关键词ABFT fault-tolerance soft errors vision transformers (ViTs) vulnerability analysis
DOI10.1109/TVLSI.2023.3317138
英文摘要Vision transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs' works mainly optimize performance and accuracy, but ViTs' reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different architecture granularities ranging from models, layers, modules, and patches for the first time. The investigation reveals that ViTs with the self-attention mechanism are generally more resilient on linear computing including general matrix-matrix multiplication (GEMM) and full connection (FC) and show a relatively even vulnerability distribution across the patches. ViTs involve more fragile non-linear computing such as softmax and GELU compared to typical CNNs. With the above observations, we propose a lightweight block-wise algorithm-based fault-tolerance (LB-ABFT) approach to protect the linear computing implemented with distinct sizes of GEMM and apply a range-based protection scheme to mitigate soft errors in non-linear computing. According to our experiments, the proposed fault-tolerant approaches enhance ViTs' accuracy significantly with minor computing overhead in the presence of various soft errors.
资助项目National Natural Science Foundation of China[62174162] ; Space Trusted Computing and Electronic Information Technology Laboratory of BICE[ETL-2022-07]
WOS研究方向Computer Science ; Engineering
语种英语
出版者IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号WOS:001086432800001
源URL[http://119.78.100.204/handle/2XEOYT63/21091]  
专题中国科学院计算技术研究所期刊论文_英文
通讯作者Liu, Cheng
作者单位1.Chinese Acad Sci, Inst Comp Technol, State Key Lab Processors, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Dept Comp Sci, Beijing 100190, Peoples R China
3.Harbin Univ Sci & Technol, Dept Comp Sci & Technol, Harbin 150006, Peoples R China
4.ASTAR, Inst High Performance Comp, Singapore 138632, Singapore
推荐引用方式
GB/T 7714		 Xue, Xinghua,Liu, Cheng,Wang, Ying,et al. Soft Error Reliability Analysis of Vision Transformers[J]. IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS,2023:11.
APA Xue, Xinghua.,Liu, Cheng.,Wang, Ying.,Yang, Bing.,Luo, Tao.,...&Li, Xiaowei.(2023).Soft Error Reliability Analysis of Vision Transformers.IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS,11.
MLA Xue, Xinghua,et al."Soft Error Reliability Analysis of Vision Transformers".IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS (2023):11.

入库方式： OAI收割

来源：计算技术研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。