Spatiotemporal Memory Network for Video Reconstruction and Super-Resolution

Abstract

Recently, as various platforms that can provide high-definition video data are provided, interest in computer vision research for video applications is increasing. Since a video contains complex information about time and space, specific strategies are required to utilize it. To utilize the information in a time series, memory modules such as recurrent neural networks (RNN) and long short-term memory (LSTM) have been used. However, since RNN and LSTM process time series through simple parameters, it is structurally difficult to preserve the visual information in the video, and the information from previous frames is lost over time. To overcome these limitations, we propose a new memory framework that can store and retrieve spatiotemporal information in a more effective way. We introduce a method for extracting and synthesizing spatiotemporal attention by extending the attention mechanism of the existing memory networks. Moreover, we introduce a method to search for adjacent spatiotemporal information to focus on important regions and reduce GPU memory consumption. Finally, through an ablation study, we find a suitable configuration for the memory component such as the sizes of channels and search regions for video tasks. We construct a video reconstruction network and train it on the Charades and HumanEva datasets to perform frame reconstruction. Furthermore, we apply the proposed structure to a super-resolution network and train it on the REDS dataset. We quantitatively compare the proposed network with existing super-resolution method, where the result shows that the proposed method achieves competitive performance.