Stochastic Learning with Back Propagation

Abstract

Despite of remarkable progress on deep learning, its hardware implementation beyond deep learning acceleration is still behind the software deep learning due in part to lack of hardware-compatible learning algorithm. In this paper, a learning method called the stochastic learning with backpropagation (SLBP) algorithm was proposed. The network of concern consists of ternary synaptic weight, favorable to be implemented in a resistance-based crossbar array. Every training epoch, the SLBP algorithm evaluates weight update probability at which the corresponding weight is updated in a stochastic manner. The algorithm was used to train a denoising autoencoder, which identified the successful reduction in noise (increase in peak signal-to-noise ratio by approximately 68%). Notably, the SLBP algorithm achieves an 86% reduction in memory usage compared with a real-valued autoencoder trained using a backpropagation algorithm.