Efficient Deep Learning via Distributed Optimization: Network Compression and Knowledge Transfer

Abstract

The success of deep neural networks (DNNs) has motivated pursuit of both computationally and memory efficient models for applications in resource-constrained systems such as embedded devices. In line with this trend, network pruning methods reducing redundancy in over-parameterized models are being studied actively. Previous works on this research have demonstrated the ability to learn a compact network by imposing sparsity constraints on the parameters, but most of them have difficulty not only in identifying both connections and neurons to be pruned, but also in converging to optimal solutions. We propose a systematic DNN compression method where weights and network architectures are jointly optimized. Additionally, we provide a holistic pruning approach, an integrated form of our method, for automatically pruning networks without specific layer-wise hyper-parameters. On the other hand, it is natural that classification models suffer from severe performance degeneration when tested on datasets different from the ones used for training. Unsupervised domain adaptation helps improve the generalizability of a pre-trained model by transferring knowledge from the labeled source domain (i.e. training set) to the unlabeled target domain (i.e. test set). The typical way of domain adaptation is to align the distributions between source and target domains. However, most existing works only enforce cross-domain consistency in the subspace while ignoring the relations of source samples to the target domain and the task-specific classifiers. Besides, first discovering a subspace shared by two domains and then training a transfer classifier separately cannot ensure whether the obtained target features are suited for the classification model. To address these issues, we propose a novel cross-domain learning method, namely robust and discriminative feature transformations (RDFT). RDFT learns domain-invariant feature representations by jointly optimizing supervised classification, unsupervised reconstruction, and distribution alignment problems. Furthermore, we introduce the concept of the weights (importance) of source instances so that the resulting classifier can be well adapted to the target domain. We solve each pruning and domain adaptation problem using alternating direction method of multipliers (ADMM), a powerful technique capable of handling non-convex separable programming. To verify the proposed joint pruning, we deployed the proposed method to a variety of state-of-the-art convolutional neural networks (CNNs) on three image classification benchmark datasets: MNIST, CIFAR-10, and ImageNet. The results show that our proposed pruning method effectively compresses the network parameters and reduces the computation cost while preserving prediction accuracy. Furthermore, experimental results on different cross-domain benchmark datasets demonstrate the encouraging performance of our RDFT over the state-of-the-art methods.