Dual Recurrent neural networks using partial linear dependence for a multivariate time series

Abstract

Multivariate time series contains two or more features observed simultaneously at each time point. The analysis of multivariate time series is still challenging due to the dynamic interrelationship that complex dependency over time and among features. This difficulty is the same for prediction. The prediction for multivariate time series has been tried a lot through the RNN series algorithm. Specifically, a structural change of the model was attempted, and new features were extracted by additionally learning neural networks such as RNN and CNN. However, when the extracted features are combined, simple concatenation is performed and learned with simple MSE, so that the dependence between features is not considered. Thus, in this paper, we propose Partial Linear Dependence for dual RNNs structure. The method considers the relationship between time-fragment multivariate time series by maximizing the partial linear dependence. And it has improved prediction performance by considering the dependencies. After the features extracted from the dual RNNs are combined, the model is trained to increase the dependency with the future fragment without overlapping each other. Therefore, it shows better performance even with the same number of parameters. Specifically, we define partial linear dependence motivated by partial least squares. Then we maximize the linear dependence between the transformed current feature fragment and the future feature fragment while minimizing the linear dependence with the current original feature fragment and its transformed feature fragment. We demonstrate the performance of the proposed method in real-life multivariate datasets, comparing it with existing models and some variants with different base cells such as RNN and GRU. We found that compared to the model without partial linear dependence in the loss function, the MSE decreased by at most 24%, and for GRU by at most 25%.