Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks
- Title
- Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks
- Author
- 최정욱
- Issue Date
- 2019-12
- Publisher
- Neural Information Processing Systems Foundation
- Citation
- Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Page. 1-10
- Abstract
- Reducing the numerical precision of data and computation is extremely effective in accelerating deep learning training workloads. Towards this end, 8-bit floating point representations (FP8) were recently proposed for DNN training. However, its applicability was only demonstrated on a few selected models and significant degradation is observed when popular networks such as MobileNet and Transformer are trained using FP8. This degradation is due to the inherent precision requirement difference in the forward and backward passes of DNN training. Using theoretical insights, we propose a hybrid FP8 (HFP8) format and DNN end-to-end distributed training procedure. We demonstrate, using HFP8, the successful training of deep learning models across a whole spectrum of applications including Image Classification, Object Detection, Language and Speech without accuracy degradation. Finally, we demonstrate that, by using the new 8 bit format, we can directly quantize a pre-trained model down to 8-bits without losing accuracy by simply fine-tuning batch normalization statistics. These novel techniques enable a new generations of 8-bit hardware that are robust for building and deploying neural network models.
- URI
- https://proceedings.neurips.cc/paper/2019/hash/65fc9fb4897a89789352e211ca2d398f-Abstract.htmlhttps://repository.hanyang.ac.kr/handle/20.500.11754/160272
- ISSN
- 1049-5258
- Appears in Collections:
- COLLEGE OF ENGINEERING[S](공과대학) > ELECTRONIC ENGINEERING(융합전자공학부) > Articles
- Files in This Item:
There are no files associated with this item.
- Export
- RIS (EndNote)
- XLS (Excel)
- XML