WLiT: Windows and Linear Transformer for Video Action Recognition
文献类型：期刊论文
| 作者 | Sun, Ruoxi1,4; Zhang, Tianzhao1,2; Wan, Yong3; Zhang, Fuping1; Wei, Jianming1 |
| 刊名 | SENSORS
 |
| 出版日期 | 2023-02-01 |
| 卷号 | 23期号:3页码:- |
| 关键词 | action recognition Spatial-Windows attention linear attention self-attention transformer |
| 英文摘要 | The emergence of Transformer has led to the rapid development of video understanding, but it also brings the problem of high computational complexity. Previously, there were methods to divide the feature maps into windows along the spatiotemporal dimensions and then calculate the attention. There are also methods to perform down-sampling during attention computation to reduce the spatiotemporal resolution of features. Although the complexity is effectively reduced, there is still room for further optimization. Thus, we present the Windows and Linear Transformer (WLiT) for efficient video action recognition, by combining Spatial-Windows attention with Linear attention. We first divide the feature maps into multiple windows along the spatial dimensions and calculate the attention separately inside the windows. Therefore, our model further reduces the computational complexity compared with previous methods. However, the perceptual field of Spatial-Windows attention is small, and global spatiotemporal information cannot be obtained. To address this problem, we then calculate Linear attention along the channel dimension so that the model can capture complete spatiotemporal information. Our method achieves better recognition accuracy with less computational complexity through this mechanism. We conduct extensive experiments on four public datasets, namely Something-Something V2 (SSV2), Kinetics400 (K400), UCF101, and HMDB51. On the SSV2 dataset, our method reduces the computational complexity by 28% and improves the recognition accuracy by 1.6% compared to the State-Of-The-Art (SOTA) method. On the K400 and two other datasets, our method achieves SOTA-level accuracy while reducing the complexity by about 49%. |
| 学科主题 | Chemistry ; Engineering ; Instruments & Instrumentation |
| 语种 | 英语 |
| WOS记录号 | WOS:000931350000001 |
| 出版者 | MDPI |
| 源URL | [http://119.78.100.198/handle/2S6PX9GI/35280]  |
| 专题 | 中科院武汉岩土力学所 |
| 作者单位 | 1.Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai 201210, China 2.School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China 3.State Key Laboratory of Geomechanics and Geotechnical Engineering, Institute of Rock and Soil Mechanics, Chinese Academy of Sciences, Wuhan 430071, China 4.School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China |
| 推荐引用方式 GB/T 7714 | Sun, Ruoxi,Zhang, Tianzhao,Wan, Yong,et al. WLiT: Windows and Linear Transformer for Video Action Recognition[J]. SENSORS,2023,23(3):-. |
| APA | Sun, Ruoxi,Zhang, Tianzhao,Wan, Yong,Zhang, Fuping,&Wei, Jianming.(2023).WLiT: Windows and Linear Transformer for Video Action Recognition.SENSORS,23(3),-. |
| MLA | Sun, Ruoxi,et al."WLiT: Windows and Linear Transformer for Video Action Recognition".SENSORS 23.3(2023):-. |
入库方式： OAI收割
来源：武汉岩土力学研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。