Transformer Encoder Attention Module

Abstract

The recent trend in video understanding utilizes self-supervised video pre-training, aiming to enable transformers to better capture the main contexts in video. Specifically, the VideoMAE (Video Masked Auto Encoder) employs a high ratio of tube masking and reconstruction as pre-training. This pre-training approach enables seamless comprehension of core characteristics in each video, resulting in remarkable performance across various video datasets. However, during the action recognition stage, the VideoMAE relies on full video frames that inherently contain temporal redundancy, leading to spatial bias in the model. To address this issue, we propose a novel module named, Transformer Encoder Attention Module (TEAM), which effectively prevents the model from being spatially biased and enhances the context modeling ability of the model. The TEAM first figures out the core features among the overall extracted features from each video. After that, it discerns the specific parts of the video where those features are located, encouraging the model to focus more on these informative parts. Consequently, our module guides the VideoMAE to effectively focus on important aspects within each video, identifying what aspects are important and where they are located. This enables it to model more accurate spatio-temporal contexts. We conduct extensive experiments to explore the optimal structure that allows for the effective integration of the TEAM with the VideoMAE. Furthermore, the integrated model (VideoMAE+TEAM) outperforms the existing VideoMAE by a significant margin on Something-Something-V2 (71.3% vs 70.3%). Finally, the qualitative results demonstrate that our module encourages the model to disregard noise and focus more on the essential video features, capturing more precise context from each video.