합성곱 신경망과 버트의 특징맵간 상관관계 분석

Abstract

With the development of deep learning, it is widely used in daily life and can be easily seen in the surrounding area. However, there are many environments where the performance of hardware is insufficient for daily life. We are interested in the pruning technology, which is one of the model compression technologies used in such situations. We conduct a feature analysis of CNN and conduct research by focusing on similar-looking feature maps obtained through correlation analysis between feature maps and logit. If the feature maps are similar, it captures the same feature, so we proceeded with the experiment assuming that neither can be eliminated. BERT Also, attention to the fact that there is a part called head that outputs the meaning of words like characteristics, We would like to devise a pruning technology that can be applied to both CNN and BERT at the same time. Prior to the weight experiment, due to the characteristics of the deep learning model, the feature map has multidimensional variables, we think that there is a limit to the commonly used correlation technology, Pearson correlation, so we decided to use a new calculating the correlation method using LMS. We tried to change the weights to measure if the LMS algorithm worked well, and gain insight into whether the correlation characteristics could apply the pruning technique to both CNN and BERT in the same way.