140 0

Convergence-Aware Neural Network Training

Title
Convergence-Aware Neural Network Training
Author
서지원
Keywords
Convergence; Training; Neurons; Correlation; Monitoring; Kernel; History
Issue Date
2020-07
Publisher
IEEE
Citation
2020 57th ACM/IEEE Design Automation Conference (DAC), page. 1-6
Abstract
Training a deep neural network (DNN) is expensive, requiring a large amount of computation time. While the training overhead is high, not all computation in DNN training is equal. Some parameters converge faster and thus their gradient computation may contribute little to the parameter update; in nearstationary points a subset of parameters may change very little. In this paper we exploit the parameter convergence to optimize gradient computation in DNN training. We design a light-weight monitoring technique to track the parameter convergence; we prune the gradient computation stochastically for a group of semantically related parameters, exploiting their convergence correlations. These techniques are efficiently implemented in existing GPU kernels. In our evaluation the optimization techniques substantially and robustly improve the training throughput for four DNN models on three public datasets.
URI
https://ieeexplore.ieee.org/document/9218518https://repository.hanyang.ac.kr/handle/20.500.11754/169267
ISBN
978-1-7281-1085-1
ISSN
0738-100X
DOI
10.1109/DAC18072.2020.9218518
Appears in Collections:
COLLEGE OF ENGINEERING[S](공과대학) > COMPUTER SCIENCE(컴퓨터소프트웨어학부) > Articles
Files in This Item:
There are no files associated with this item.
Export
RIS (EndNote)
XLS (Excel)
XML


qrcode

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

BROWSE